JMIR Medical Informatics
Volume 6 (2018), Issue 3

Impact Factor (2023): 3.1
ISSN 2291-9694 Editor in Chief: Christian Lovis, MD, MPH, FACMI

Contents
Original Papers
Extraction and Standardization of Patient Complaints from Electronic Medication Histories for
Pharmacovigilance: Natural Language Processing Analysis in Japanese (e11021)
Misa Usui, Eiji Aramaki, Tomohide Iwao, Shoko Wakamiya, Tohru Sakamoto, Mayumi Mochizuki. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

Uncovering a Role for Electronic Personal Health Records in Reducing Disparities in Sexually Transmitted
Infection Rates Among Students at a Predominantly African American University: Mixed-Methods Study(e41)
Kevon-Mark Jackman, Stefan Baral, Lisa Hightow-Weidman, Tonia Poteat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

Defining Empowerment and Supporting Engagement Using Patient Views From the Citizen Health
Information Portal: Qualitative Study (e43)Tracie Risling, Juan Martinez, Jeremy Young, Nancy Thorp-Froslie. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

Emergency Physician Use of the Alberta Netcare Portal, a Province-Wide Interoperable Electronic Health
Record: Multi-Method Observational Study (e10184)
Timothy Graham, Mark Ballermann, Eddy Lang, Michael Bullard, Denise Parsons, Gabriella Mercuur, Pat San Agustin, Samina Ali. . . . . . . . . . . . . .

40

Task-Data Taxonomy for Health Data Visualizations: Web-Based Survey With Experts and Older Adults(e39)
Sabine Theis, Peter Rasche, Christina Bröhl, Matthias Wille, Alexander Mertens. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

53

Implementing a National Electronic Referral Program: Qualitative Study (e10488)
Marcella McGovern, Maria Quinlan, Gerardine Doyle, Gemma Moore, Susi Geiger. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

73

Validating a Framework for Coding Patient-Reported Health Information to the Medical Dictionary forRegulatory Activities Terminology: An Evaluative Study (e42)
Sonja Brajovic, David Blaser, Meaghan Zisk, Christine Caligtan, Sally Okun, Marni Hall, Carol Pamer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

89

ReviewThree-Dimensional Portable Document Format (3D PDF) in Clinical Communication and Biomedical
Sciences: Systematic Review of Applications, Tools, and Protocols (e10295)
Axel Newe, Linda Becker. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

98

JMIR Medical Informatics 2018 | vol. 6 | iss. 3 | p.1

XSL• FO
RenderXJMIR MEDICAL INFORMATICS

Usui et al

Original Paper

Extraction and Standardization of Patient Complaints from
Electronic Medication Histories for Pharmacovigilance: Natural
Language Processing Analysis in Japanese
Misa Usui1, BA; Eiji Aramaki2, PhD; Tomohide Iwao2, MS; Shoko Wakamiya2, PhD; Tohru Sakamoto3, BA; Mayumi
Mochizuki4,5, PhD
1Division of Hospital Pharmacy Science, Graduate School of Pharmaceutical Sciences, Keio University, Tokyo, Japan

2

Social Computing Lab, Graduate School of Information Science, Nara Institute of Science and Technology, Nara, Japan

3

Holon Co, Ltd., Hiroshima, Japan

4

Division of Hospital Pharmacy Science, Faculty of Pharmacy, Keio University, Tokyo, Japan

5

Department of Pharmacy, Keio University Hospital, Tokyo, Japan

Abstract
Background: Despite the growing number of studies using natural language processing for pharmacovigilance, there are few
reports on manipulating free text patient information in Japanese.
Objective: This study aimed to establish a method of extracting and standardizing patient complaints from electronic medication
histories accumulated in a Japanese community pharmacy for the detection of possible adverse drug event (ADE) signals.
Methods: Subjective information included in electronic medication history data provided by a Japanese pharmacy operating in
Hiroshima, Japan from September 1, 2015 to August 31, 2016, was used as patients’ complaints. We formulated search rules
based on morphological analysis and daily (nonmedical) speech and developed a system that automatically executes the searchrules and annotates free text data with International Classification of Diseases, Tenth Revision (ICD-10) codes. The performanceof the system was evaluated through comparisons with data manually annotated by health care workers for a data set of 5000
complaints.
Results: Of 5000 complaints, the system annotated 2236 complaints with ICD-10 codes, whereas health care workers annotated2348 statements. There was a match in the annotation of 1480 complaints between the system and manual work. System performance
was .66 regarding precision, .63 in recall, and .65 for the F-measure.
Conclusions: Our results suggest that the system may be helpful in extracting and standardizing patients’ speech related tosymptoms from massive amounts of free text data, replacing manual work. After improving the extraction accuracy, we expect
to utilize this system to detect signals of possible ADEs from patients’ complaints in the future.
(JMIR Med Inform 2018;6(3):e11021) doi:10.2196/11021
KEYWORDS
adverse drug events; natural language processing; medical informatics; medication history; pharmacovigilance

Introduction
Background
Adverse drug events (ADEs) are any untoward injuries resulting
from the use of a drug [1]. They occur in around 18% of
inpatients [1-4] and are a significant burden on health care andsociety. The ADEs are a cause of morbidity, and mortality andtheir economic loss is estimated at US $177.4 billion annuallyin the US [5]. In the field of pharmacovigilance, postmarketingsurveillance such as spontaneous reporting is important for thedetection of ADEs because clinical trials have limitations
including patient sample size, population, and administration
period [6].
http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

The need to understand patients’ subjective complaints and touse other sources in pharmacovigilance has increased. Unlike
health care providers, patients use various expressions andterminology to describe their situations. Direct reporting frompatients is helpful in understanding their detailed symptoms andimpacts on quality of life, which medical professionals tend tooverlook [7-9]. For example, analysis of the content ofcomments posted on patients’ online community pages revealed
unknown long-term symptoms of antidepressant withdrawal[10]. The Maintenance and Support Services Organization
developed the Patient-Friendly Term List [11] based on themost frequent ADEs reported by patients and consumers tofacilitate direct patient reporting of ADEs to regulators and thepharmaceutical industry. Despite its importance, little work has
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.2(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

been done on exploring patient records until recently due totheir unstructured, time-consuming data format.

Methods

Natural language processing (NLP) is the automatic
manipulation of natural language such as narrative text andspeech for extraction and structuring [12]. Numerous attempts
have been made to use NLP in electronic health records (EHRs),social media, medical literature, or existing reporting systems[13-29]. Those studies found that NLP could identify various
points for the assessment of medications (eg, inactive
medication, nonadherence, patients’ mentions of ADEs).

Concept of the System

In Japan, text analysis and automated detection of medical
events from EHRs have been reported [30], and a tool for
disease entity encoding was developed [31]. However, these 2
studies intended only to manipulate clinical text provided byhealth care professionals using medical terminology. No
previous study dealt with patients’ complaints in their own
words in Japanese.

Prior Work
Nikfarjam et al [25] introduced a machine learning-based
extraction system using conditional random fields (CRFs) for
user posts on DailyStrength (precision: .86, recall: .78,
F-measure: .82) and Twitter (precision: .76, recall: .68,
F-measure: .72) to detect adverse drug reaction (ADR) signals.
Freifeld et al [28] classified Twitter posts (precision: .72, recall:.72) to compare product-event pairs with the US Food and Drug
Administration Adverse Event Reporting System (FAERS) data.
In the mining of patients’ reports, Topaz et al [26] used alinguistic-based approach comparing EHRs (clinicians’ reports)and social media (patients’ mentions) for 2 common drugs.
White et al [27] used search log data for the identification of
ADE signals and a comparison with FAERS data resulted inhigh concordance as determined by the Area Under the Curve
Receiver Operating Characteristics curve of .82. Denecke et al[29] collected data from multiple media sites with keyword lists
and classified texts as relevant/irrelevant using support vector
machines.
Although no previous studies have been completed in Japanese,
Aramaki et al [30] reported on a system to extract medical event
information from Japanese EHRs based on CRFs (precision:.85, recall: .77, F-measure: .81). The text source in their study
was written in medical terminology, mainly by physicians. No
lexicon to standardize patients’ informal expressions such as
the Patient-Free Term List [11] and the work of Freifeld et al[28] has been published in Japanese.

Study Aim
This study aimed to develop techniques to establish a method
for extracting and standardizing patient complaints fromelectronic medication history data (EMHD) accumulated in a
Japanese community pharmacy for the detection of possible
ADE signals.

We propose a system that automatically extracts andstandardizes patient complaints (Figure 1). In this system,
subjective information included in the medication histories
collected from a pharmacy is input data, and data in which
International Classification of Diseases, Tenth Revision(ICD-10) codes are attached to patient expressions are outputs.
A dictionary-based method was adopted for extraction andstandardization. The processing steps in the system are as
follows. First, morphological analysis is performed on input
data. Next, the search rules are applied to split data. In the searchrules, morpheme combinations in general expressions and thecorresponding ICD-10 codes are described for each line, andexclusion rules are set for some ICD-10 codes. When a patient
expression satisfies the search rules, a corresponding ICD-10
code is given. Procedures for creating the search and exclusion
rules and system development procedures are detailed in “Search
Rules” and “System Development.”

Data Sources
The EMHD stored in a community pharmacy were used as thesource of patients’ comments. When pharmacists dispense
prescription drugs to patients, they are required to record theresults of medication instructions and patients’queries/responses. A medication history in Japan is typically
written in the “SOAP” format, which consists of 4 sections:“Subjective information” (complaints of the patient), “Objective
information” (objective indicators such as laboratory findings
or names of drugs prescribed), “Assessment” (the pharmacist’s
findings on the occurrence of ADRs, interactions, or doubt aboutprescription instructions), and “Plan” (action plan of thepharmacist derived from the assessment).
Although patients do not write the medication history, of those
4 sections the “Subjective information” appeared to be the mostappropriate text source, because pharmacists complete that
section in the patients’ own words.
Patients’ comments were extracted from the EMHD of acommunity pharmacy operated by Holon Co, Ltd, Hiroshima,
Japan. This company operates a chain of 14 pharmacies, andthe data used in this study mainly came from a single one. The
study period was from September 1, 2015 to August 31, 2016.
Personal information such as patients’ names and birth dates
were anonymized before analysis.
Information on the hospitals or clinics that issued prescriptions
for which the subjective information used in this study wasderived is shown in Table 1. The pharmacy filled a total of42,120 prescriptions during the study period for the top 9
prescribing hospitals or clinics. The number of prescriptions
from medical institution A was the highest (18,273/42,120,
43.5%). Clinic A specializes in otolaryngology, and the patientsare older adults who often complain of dizziness or hearing loss.
Table 2 shows the items recorded in the EMHD, while Figure
2 is an example of a recording object.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.3(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Figure 1. Concept of the system. ICD-10: International Classification of Diseases, Tenth Revision..

Table 1. Backgrounds of the prescriptions used in this study (N=42,120).
Hospital/ clinic

Specialty

Main patient characteristics

Prescriptions, n (%)

A

Otolaryngology

•
•
•
•

ElderlyDizziness
Tinnitus
Hearing loss

18,273 (43.38)

B

General medicine; cardiology•••

Elderly
Heart disease
Hypertension

5356 (12.72)

C

General medicine; gastroenterology; cardiology •••Elderly
Digestive tract diseases
Circulatory diseases

537 (1.27)

D

General medicine; Kampoa medicine••

Elderly989 (2.35)

E

Neuropsychiatry•

Wide range of age-groups

377 (0.90)

F

Neuropsychiatry•

Wide range of age-groups

563 (1.34)

G

Obstetrics and gynecology•

Gynecological conditions

649 (1.54)

H

Obstetrics and gynecology•

Infertility treatment

4206 (9.99)

I

Breast surgery•

Breast cancer patients visiting for diagnosis andpostoperative care

2608 (6.19)

Otherb

Various•

Various8562 (20.33)

Kampoa medicines (for >half of the patients)a

The word “Kampo” means herbal medicine in Japanese. The term “Kampo Shoseiryuto” is commonly used to treat watery nasal discharge, nasal
congestion, watery sputum, and sneezing.
b

These are medical institutions that are not the major clinics “A” to “I” from which this pharmacy receives prescriptions.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.4(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Table 2. Items recorded in the electronic medication history data.
Category

Content

Identification

Identification of patient

Name

Name of patient

Sex

Male/female

BirthaYear/month/day

Special instructions

Notes as required (eg, disease entity)

Concomitant drugs
Drug name

YJ-codeb or product name

Dosing period

Year/month/start date/end datePrescription data
Dispensing dateYear/month/day

Drug name

YJ-code or product name

Dosage/administrationc

Free text

Medication counselingaCounseling dateYear/month/day

Subjective informationd

Free text

Objective informatione

Free text

Assessmentf

Free text

Plang

Free text

Only the year of birth data was used.

b

YJ-codes identify prescription drugs covered by insurance in Japan.

c

Not used in this study.

dIncludes patient complaints.

e

Includes laboratory data.

fIncludes pharmacists’ assessments of patient conditions.

gIncludes pharmacists’ plans for prescription questions, patient education, and follow-up.

Figure 2. Example of a medication history.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.5(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Search Rules
We created search rules to identify the appropriate ICD-10 codefrom the free text in the “Subjective information” section anddeveloped a coding system that annotates the ICD-10 codes
within patient complaints. The ICD-10 was originally an
English-based system but is also used in Japan. It was translated
into Japanese by the World Health Organization, and a coding
rulebook was published. For example, in Medis [32] the ICD-10
is given as the basic classification code, and coding matched as
closely as possible to clinical interpretation is undertaken.
Although it may be possible to use the Medical Dictionary forRegulatory Activities (MedDRA) or the International
Classification of Primary Care as a medical code system, we
adopted ICD-10 in this study because it is used for insurance
claims in Japan and because many coders are familiar with
ICD-10.
In developing the system, a nurse with 10 years of experiencein the field of terminal care and a medical coder with 20 yearsof experience created the search rules based on the expressions
in the “Subjective information” section. A programmer readthe search rules and developed a program to accommodate new
expressions. Search rules were created by a combination ofmorphological analysis and common expressions.
The search rules govern the pattern for analyzing comments
included in the subjective information. The rules were saved in
Microsoft Excel format with the corresponding disease entity
category and ICD-10 codes. For example, to search for “D69.9:
Hemorrhagic condition, unspecified,” the search strings are “(出血|しゅっけつ|血)+(傾向|けいこう|し易く|しやすく|し易
い|しやすい|出やすい|止まりにくい|とまりにくい|止まらない|とまらない).” In English, this would translate to“(bleeding|blood)+(tendency|easy to|hard to stop|won’t stop|notstop).” Written Japanese utilizes 3 orthographic systems:
Chinese characters, hiragana, and katakana. Therefore, theactual search strings are longer than in English. All rules areshown in Multimedia Appendix 1. The rule-making steps areshown in Textbox 1. We repeated this process 5 times over 1
month in order to refine the search rules.
The nurse first checked the free text recorded in the “Subjective
information” section and selected complaints referring topatients’ symptoms. Then words related to ICD-10 codes weremanually extracted from the complaints. Finally, the extracted
words were added sequentially to the search string for each
ICD-10 code. The search strings consist of patterns of word
combinations using “|” (logical sum) or “+” (logical product).
At present, a maximum of 3 words/terms can be combined in

Usui et ala string separated by “+” signs. For example, from the text
“blood pressure today was a little high,” the terms “bloodpressure” and “a little high” were extracted, and the systemannotated the text with the ICD-10 code “I10: hypertension.”
However, some text found in the “Subjective information”
section could not be annotated with an ICD-10 code even though
it followed the search rules. Therefore, we set exclusion rules
for some codes, which were created following the sameprocedure as for the search rules but were only applied when ahealth care worker could visually confirm the keyword for
exclusion. For the previous example of “D69.9: Hemorrhagic
condition, unspecified,” terms with “(-|ない|なし|無い|無し),”
in English, “(-|no|none|negative|never|don’t)” were excluded
even if they included search strings. For example, “(血がとまりにくい),” in English, “(the bleeding won’t stop),” wasannotated as D69.9, but “(血がとまりにくいことはない),”in English, “(I never felt the bleeding wouldn’t stop),” wasexcluded.

System Development
The system developed extracts complaints related to patients’
symptoms from the “Subjective information” section of EMHD
automatically and annotated each complaint with the ICD-10
code using the search rules above. During system development,
we used Perl as the programming language and MeCab [33] asa morphological analyzer. The Microsoft Excel format was used
for subsequent analysis.
The development procedure can be summarized as follows:
1.
2.

3.

4.

Subjective information was extracted from each saved
Microsoft Excel file
Morphological analysis was performed to extract subjective
information, separating the text with spaces into minimum
meaningful units of words/terms
After the processes above were performed, the subjective
information was copied back into a Microsoft Excel file.
Search rules and exclusion rules were applied to thesubjective information by analyzing each complaint andsearching for the ICD-10 code
If an appropriately matching ICD-10 code was found, thecomplaint was annotated with the ICD-10 code and thecorresponding disease entity

The coding system adapts the search rules (shown in Multimedia
Appendix 1) in order from the top. If an adaptable rule is found,
the result of ICD-10 coding is output. If multiple rules arematched, all of them are output in the results.

Textbox 1. Rule-making steps.
1.

Make seed rules

2.

Apply seed rules to development set

3.

Error analysis performed by two rule curators (the nurse and medical coder)4.

New rules added by the programmer, who converts the error analysis to general expressions

5.

Repeat steps 3 and 4

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.6(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Figure 3. System interface screenshot. ICD-10: International Classification of Diseases, Tenth Revision.

Optimization of System Performance
For optimal performance of the system, the system-annotateddisease entities should ideally match the entities manuallyannotated by health care professionals. As mentioned above,the more thoroughly the search rules are satisfied, the moreaccurate the system. Therefore, we reviewed the search rules
multiple times to determine the most appropriate ones toimprove the accuracy of the system.
In this study, we did not attempt machine learning for thedetection of relevant terms to match ICD-10 codes. By adding
search rules as appropriate, free text can be automaticallyassociated with ICD-10 codes via the system.

Experiment
An evaluation experiment was conducted to confirm theperformance of the system. Five thousand complaints from thesubjective information were processed, and 323 search rules
were created. In the experiment, health care workers (1 nurse
and 1 pharmacist) first independently annotated the 5000
complaints manually with the ICD-10 codes. Second, 108
mismatched annotations were excluded, and the data from theremaining 2348 were used as correct answers for the subsequent
step. Finally, the system with 323 search rules was applied to
the 5000 complaints.
The subjective information used in this study consisted ofmultiple sentences, and thus several patient expressions wereobtained from one “Subjective information” section. Since eachpatient expression is linked to the ICD-10 code, multiple ICD-10
codes are assigned to a single “Subjective information” section.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

In evaluating the system in this study, if one of the plural ICD-10
codes differed from the manual result, it was judged that allother coding for that entry was incorrect (unmatched). Figure
3 shows an actual system execution screen.
Based on the results of this experiment, the precision, recall,and F-measure of the system were calculated [34,35]. Precision
was calculated by dividing the matched number (the number of“Subjective information” sections for which manual coding andsystem coding had the same result) by the searched number (thenumber of “Subjective information” sections that the systemannotated with ICD-10 codes). Recall was calculated by dividing
the matched number by the correct answers (the number of“Subjective information” sections manually coded). The
F-measure was calculated by taking the harmonic mean between
precision and recall.

Ethical Considerations
This study was approved by the Ethics Committees on Human
Research of the Faculty of Pharmacy, Keio University and Nara
Institute of Science and Technology.

Results
Examples of correct answer data and system execution results
are shown in Table 3. From 5000 complaints, 2348 ICD-10
codes were extracted by health care workers. The system
extracted 2236 codes, 1480 of which matched the manual results.
The system performed .662 for precision, .630 for recall and.646 for F-measure. Table 4 shows precision and recall for the
10 most frequent symptoms extracted by health care workers.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.7(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Table 3. Comparison between manual and system extraction of ICD-10 codes for patient complaints from typical examples.
Patient complaints (Original text of Japanese is attached)

Manual results

めまい は 起こっ て い ない です 。 暑 さ で フラー っと する R42: Dizziness and giddiこと が ある 。
ness; R26.0: Ataxic gait

System results

Matching

R42: Dizziness and giddinessNot matched

J00: Acute nasopharyngitis(common cold); R51: Headache

MatchedNo vertigo/dizziness. Sometimes I feel unsteady due to the heat.
昨日 の 夜 から 頭 の 後ろ が 重い よう な 感じ が する 。 風邪 J00: Acute nasopharyngitis
か と 思っ て 受診 。
(common cold); R51:
From the night before last, the back of my head felt heavy. I probably Headachecaught cold, so I saw a doctor.
私 よく 走っ ちゃう ん だ けど 坂 で 躓い て ね 。 ちょっと な Q82.5: Congenital nonneoのに 大きな アザ ガ でき た 。 整形 外科 の 先生 は 血 流 の 薬 plastic nevus
が ある から って 。 で も 飲ま ない と だめ だ と いわ れ た。

Q82.5: Congenital nonneoplastic Matchednevus

I was in a hurry, stumbled, and fell. A big bruise came up. My orthopedist said it may be caused by my blood thinner, though I have tocontinue that medicine.
血圧 は ちょっと 高かっ た の 。 １ ４ ０ 後半 。 家 だっ たら I10: Essential (primary) hy- I10: Essential (primary) hyperten- Matched
１ ３ ０ くらい なん だ けど 。 整形 外科 で もらっ た パップ pertension; L259: Unspeci- sion; L259: Unspecified contact
は かぶれ て しまっ た 。
fied contact dermatitis, un- dermatitis, unspecified cause
specified cause
My blood pressure was a bit high, in the upper 140s, although it’saround 130 when I’m at home. I got a rash from plasters prescribedby orthopedics.
風邪 を 引き まし た 。 この間 、 ヘルペス の 薬 を X月Y日 か J00: Acute nasopharyngitis;
ら 飲ん で X’月 Y’日くらい から ２ ． ３ 日 、 夜 の トイレ で R32: Unspecified urinary
尿 が 我慢 でき ず に もらし そう な こと が 続い て しんどかっ incontinence
た 。 あんな こと は 初めて 。 薬 情 を 見 たら 腎臓 に 悪く なる と 書い て ある から この 薬 の せい だ と 思う 。
I caught a cold. The other day I started taking an antiherpes prescription from mm/dd, and few days afterward I felt like I was going to
pee in my pants and was really exhausted. I never had such an experience. Looking at the drug information, this medicine may have bad
effects on the kidney. Maybe it’s due to this medicine.

J00: Acute nasopharyngitis;
Not matched
B02.9: Zoster without complication; R32: Unspecified urinary
incontinence; R53: Malaise andfatigue; N28.9: Disorder of kidney and ureter, unspecified;
R94.4: Abnormal results of kidney function studies

Table 4. Precision and recall for the 10 most frequent symptoms.

a

Rank

ICD-10aMatched, n (%)

Correct answer, n (%)

Searched, n (%)

Precision

Recall1

R42 (Dizziness and giddiness)

146 (9.86)

173 (7.37)

254 (11.36)

.575

.844

2

J00 (Acute nasopharyngitisb)

189 (12.77)

208 (8.86)

226 (10.11)

.836

.909

3

R52.9 (Pain, unspecified)

108 (7.29)

146 (6.22)

199 (8.90)

.543

.740

4

F19.6 (Mental and behavioral disordersc)

142 (9.59)

168 (7.16)

197 (8.81)

.721

.845

5

R05 (Cough)114 (7.70)

134 (5.71)

143 (6.40)

.797

.851

6

H931 (Tinnitus)

99 (6.68)

131 (5.58)

134 (6.00)

.739

.756

7

R26.0 (Ataxic gait)

41 (2.77)

59 (2.51)

120 (5.37)

.341

.695

8

I10 (Essential [primary] hypertension)

51 (3.44)

89 (3.79)

87 (3.89)

.586

.573

9

F51.1 (Nonorganic hypersomnia)

39 (2.63)

46 (1.96)

85 (3.80)

.459

.848

10

R53 (Malaise and fatigue)

40 (2.70)

59 (2.51)

81 (3.62)

.494

.678

ICD-10: International Classification of Diseases, Tenth Revision.

bCommon cold.

c

Due to multiple drug use and use of other psychoactive substances.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.8(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Textbox 2. Six reasons for unmatched results. ICD-10: International Classification of Diseases, Tenth Revision.
1. Misdetection of negation or possible event
System misread an expression including negation or possible event as a symptom that actually occurred (eg, “dizziness has not occurred,” “If I feel
dizzy”)
2. Misdetection of a clinical test item
System mistook a clinical test term as a patient symptom (eg, “test for dizziness”)
3. Misdetection of drug class name
System mistook the name of the drug class as a patient symptom (eg, “painkiller” mistaken for “R529: Pain, unspecified”)
4. Misdetection of unrelated words
System mistook unrelated words as a patient symptom (eg, “I’m getting old” mistaken for “R54: Senility”)
5. False negative
System missed a word that indicates a patient symptom
6. Inappropriate ICD-10 code
System failed to choose the appropriate ICD-10 code even if it extracted words related to a patient symptom

The results indicated that the average performance of the system
was .66 for precision, .63 for recall, and .65 for the F-measure.
Comparing the performance for each symptom, the precisionof “dizziness and giddiness,” “pain, unspecified,” and “ataxicgait” was especially low. We identified 6 reasons for theunmatched results for these 3 symptoms, as shown in Textbox2.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

Table 5 details the unmatched results and typical examples for
3 symptoms. The main reason for discordance between manualand system coding was misdetection of negation or possible
event in “R42: dizziness and giddiness” (79/108 results, 73.1%)
and “R26.0: ataxic gait” (71/79 results, 90%), whereas
misdetection of drug class name was the most common in“R52.9: pain, unspecified” (28/91 results, 31%).

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.9(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Table 5. Details of unmatched results and typical examples for 3 symptoms.
Symptom and categoryn (%)

Example of Patient complaints (Original text of Japanese is attached)

Manual results

System results眩暈 は 起き て い ない が 、 ２ 回 くらい ふらつき が あっ R26.0: Ataxic
た 。 今日 Ｄｒ に 診 て もらっ たら 大丈夫 だ と 言わ れ gait
た。

R42: Dizzinessand giddiness;
R26.0: Ataxic
gait

R42: Dizziness and giddiness (N=108)
1 (Misdetection of nega- 79 (73.1)
tion or possible event)I didn‘t feel dizzy but staggered about twice. Today I saw a doctor
and be said there was no problem.
2 (Misdetection of clinical test item)

5 (4.6)

なぜ か ニセルゴリン だけ １ ０ 錠 余っ てる 。 調子 は—
なんとも ない 。 眩暈 も 検査 し た けど 問題 ない と 言
わ れ た し 、 聞こえ も 悪く なっ て ない と 言わ れ た 。
I don’t know why, but I have 10 leftover Nicergoline pills. I feelfine. I was examined for dizziness and no problem was found. I
was also told that there was no problem with my hearing.

3 (Misdetection of drugclass name)8 (7.4)

めまい の 薬 は 昼 飲め ない こと が 多く て 残り が あるの 。 ずっと 仕事 だ から 飲む の が 難しく て 。

—

I have leftover motion sickness medicine because I often don’ttake it in the daytime. I work all day and it’s hard to take it.
4 (Misdetection of unrelated words)

4 (3.7)

母 の 介護 で 忙しかっ た けど 秋 に 亡くなっ て 、 葬式 R42: Dizzinessとか で また 忙しく て あまり 薬 が 飲め て なかっ た の and giddinessで 残り は あり ます 。 めまい も そんなに ひどく なっ てなかっ た です 。
I was busy taking care of my mother. She passed away this autumn and I got busy with the funeral arrangements, etc. I didn’t
have time, so there are still some pills. The dizziness didn’t get
much worse.

R42: Dizzinessand giddiness;
H91.9: Unspecified hearing loss

R42: Dizzinessand giddiness;
R13:Dysphagia

R42: Dizzinessand giddiness;
R99: Other ill-defined and unspecified causes ofmortality

5 (False negative)

11 (10.2)

めまい は 起こっ て い ない です 。 暑 さ で フラー っとする こと が ある 。

R42: Dizzinessand giddiness;
No vertigo/dizziness. Sometimes I feel unsteady due to the heat. R26.0: Ataxic
gait

R42: Dizzinessand giddiness6 (Inappropriate ICD10code)

1 (0.9)ふらふら する の は 最近 落ち着い てる 。 Ｄｒ に 言わ れ R42: Dizzinessて 、 起き上がる とき も ゆっくり 起きる よう に し たら and giddiness;
眩暈 あまり 起き なく なっ た 。
H81.1: Benign
paroxysmal vertiThe dizziness is getting better. My doctor told me to stand upgoslowly and when I tried that I didn’t feel dizzy.

R42: Dizzinessand giddiness;
R26.0: Ataxic
gait

R52.9: Pain, unspecified (N=91)
1 (Misdetection of nega- 20 (22.0)
tion or possible event)血圧 １ １ ０ くらい 。 ふらつく こと は ない です 。 筋 R53: Malaise and M79.1: Myalgia;
肉 痛 も ない です 。 気 に なっ た こと は ない が 、 言わ fatigue
R52.9: Pain, unれ て みれ ば 、 たまに だるく なる こと が あり ます 。
specified; R53:
Malaise and faBP is around 110. No dizziness and no muscle pain. But now
tigue
that you ask, I sometimes feel malaise.

3 (Misdetection of drugclass name)

痛く ない から 神経 痛 の 薬 は もう いら ない 。 納豆 が —本当は 大好き な ん だ けど 、 ワーファリン 錠 飲ん でるから 止め てる 。 納豆 関係 ない 薬 も ある らしい けど、 値段 が 高い らしい ね 。

28 (30.8)

II don’t feel pain, so I don’t need the nerve medicine any longer.
I really love natto (fermented soybeans), but have to avoid it because I take warfarin. I think there is some kind of medicine thatis not affected by eating natto, but I hear it’s really expensive.
4 (Misdetection of unrelated words)

8 (8.8)

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

耳鳴り と のど が 痛い 。
My ears are ringing and I have a sore throat.

H93.1: Tinnitus;
J02.9: Acutepharyngitis, unspecified

R52.9: Pain, unspecified; M79.2:
Neuralgia andneuritis, unspecified

H92.0: Otalgia;
H92.1: Otorrhea;
R52.9: Pain, unspecified

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.10
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Symptom and categoryn (%)

Example of Patient complaints (Original text of Japanese is attached)

Manual results

5 (False negative)

9 (9.9)

胃 が 痛い 、 のん で も 治ら ん から 先生 に 相談 し た 。
腰 も い たい ん よ ね 。 ロキソニン は 手持ち 無くなった 。 今回 で て ない ？ あら 。 ボラザ Ｇ 軟膏 は ２ １ 本
持っ てる けど ちょっと 足り ない かも ね 。

R10.1: Pain localized to upper abdomen; R52.9:
Pain, unspecified;
M54.5: Low backMy stomach hurts. I took some medicine but didn’t feel betterso I saw a doctor. My lower back also hurts. I’m out of Loxopro- pain
fen, so did the doctor give me a new prescription? I have 21 tubesof ointment but that may not be enough.

R101: Pain localized to upper abdomen; R52.9:
Pain, unspecified

6 (Inappropriate ICD10code)

26 (28.6)

前 の 粉薬 を 止め たら 尿 の 出 は 治っ た ん だ けど 、他 の 薬 を 続け て い たら 手の甲 と 関節 が ピリピリ 痛
み 始め て 驚い た 。 飲む の を 辞め たら 治っ た よ 。

M25.5: Pain injoint; 52.9: Pain,
unspecified

M25.5: Pain injoint; M79.6:
Pain in limb

System results

After I stopped taking the powdered medicine, my urine flow
improved but I’m still taking the other medicine. I suddenly felt
a sharp pain on the back of my hand and in the joints, but that
stopped after I quit taking the medicine.
R26.0: Ataxic gait (N=79)
1 (Misdetection of nega- 71 (89.9)
tion or possible event)

カルデナリン 錠 が 中止 に なっ て 、 フラ ツキ は あれ—
っきり よく なっ た 。 血圧 の 薬 は 内科 で かるい の が追加 に なり まし た 。 眼科 は 今 は かかっ て なく 、 内科 で 目薬 出し て もらっ て ます 。

R26.0: Ataxic
gait

After I stopped taking Doxazosin, the light-headedness got better.
A new mild blood pressure medicine was prescribed by the doctor. I don’t go to the eye doctor anymore, so I have my regular
doctor prescribe eyedrops.
4 (Misdetection of unrelated words)

2 (2.5)

かるい フラ ツキ ある ん だ けど 大丈夫 。 内科 は いって ない 。 先生 行け って 言わ ない から 大丈夫 な ん だと 思う 。 セロクラール と パルトックス 錠 は わかる から 赤 線 は ひか なく て いい よ 。

R26.0: Ataxic
gait

I stagger a little bit, but that’s OK. I didn’t go to the Internal
Medicine Department because my doctor didn’t tell me to, so it
probably isn’t a problem. I know which pills are Cerocral andPantethine, so you don’t have to draw a red line on the bottles
for me.
5 (False negative)

5 (6.3)血圧 は いつも より 良かっ た 。 病院 で 血糖 が ６ ８ だっ E16.2: Hypoglyた から 飴 玉 食べ た 。 ブドウ糖 も 持っ てる 。 ふわっ caemia, unspeciと する 症状 も あっ た 。
fied; R26.0:
My blood pressure was better than usual. My blood glucose was Ataxic gait

R26.0: Ataxic
gait; R21: Rashand other nonspecific skin eruption

R26.0: Ataxic
gait

68, so I ate some candy. I carry glucose tablets with me, too. I
felt weightless.
6 (Inappropriate ICD10code)

1 (1.3)

頭 が フラフラ する ので 医師 に 相談 し た 。 リリカ が効き 過ぎ て いる の で は ない か と の こと 。

R42: Dizzinessand giddinessR26.0: Ataxic
gait

I felt woozy and went to the doctor. He thought that the Pregabalin was too strong.

Discussion

are at least adequate as the first step in possible ADE signal
detection.

Principal Results

This was the first attempt to standardize patients’ expressions
with the Japanese version of ICD-10 and to use the “Subjective
Information” section in the medication history as a source. The
advantage of using the medication history is its structured format
and data storability. The medication history is recorded forpatient monitoring including side effects. Its features provide
more specialized information relevant to possible ADEs thansocial media like Twitter or EHRs in hospitals. Moreover, thenumber of pharmacies in Japan is increasing [36,37], andpharmacists are required to record patient medication histories
for health insurance claims. Thus, huge amounts of data on

Nikfarjam et al [25] and Aramaki et al [30] used CRFs, andFreifeld et al [28] used a tree-based dictionary matchingalgorithm for extracting the terms. Our approach involved
rule-based searching, which is much simpler but less tolerantof orthographic variants. Additionally, differences in linguisticfeatures might have contributed to the gap between the resultsof the present study and nonJapanese ones [25,28]. In written
Japanese, words are not separated by spaces, and therefore theaccuracy of extraction is affected by the quality ofmorphological analysis. Considering these points, the results
http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.11
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
patients’ medication are available, making medication histories
an appropriate source for ADE signal detection.
It is not necessary for ADEs to have causal relationships withdrugs, whereas ADRs must have a reasonable association withdrug use. Using patient records to detect ADRs is a major
challenge because causality cannot be readily assessed; however,
it is also important to detect potential ADE signals.
In this study, some text could not be annotated with ICD-10
codes. As compared with the performance of health care
professionals, our newly developed system performed at levelsof .66 for precision, .63 for recall, and .65 for the F-measure.
These values are relatively lower than in previous studies[25,28,30], likely due to differences in methodology. As
explained in the Experiment section, if one of the many ICD-10
codes was different from the manual result, all other coding was
regarded as incorrect (unmatched) for that entry. This is one
reason why the F-measure was lower than in previous research.
There was also insufficient specific information about thecondition of each patient. Because the majority of patients are
not medical experts, they describe their symptoms in everyday
language, which is more equivocal and more inflected than
medical terminology. Nikfarjam et al [25] reported similar
aspects of ambiguity and lack of context in patients’ wording.
The dialect spoken can affect the subjective information,although, of 5,000 complaints analyzed in this study, only 7
were recorded in a regional dialect. This is probably related to
the nature of the text. Although it is recommended thatpharmacists record patients’ statements exactly, it is possiblethat they replace dialect expressions with standard wording to
make the information easier to understand by others later.
Regarding standardization across languages, the present system
could be applied to other languages to some extent by translating
the morphemes used for the search rules or by adding or refining
the search rules later.

Limitations
There were some limitations of this study. First, qualitative
differences in the text data could have occurred. The “Subjective
information” section is filled in by pharmacists, and therefore
they may interpret and summarize patients’ comments when
they record them. To ensure that the medication histories of all
patients are recorded during the daily business hours ofcommunity pharmacies, in some cases fixed-form complaint
set phrases and excerpts of comments may be relied on to
decrease the time needed to complete the “Subjective
information” section. It is therefore possible that the finer

Usui et alnuances patients hope to convey are altered or lost during theprocess. Qualitative differences were also noted among
pharmacists for the contents of the “Subjective information”
section. Some wrote about symptoms using explicit medical
terminology (eg, “back pain and knee pain were unabated”).
Others included general information unrelated to symptoms (eg,greetings and general conversation transcribed word for word).
Second, it was difficult for the system to determine whether the
extracted keyword was related to patients’ symptoms or thoseof others. For example, from the sentence “My friend had
hypertension,” the system may extract “hypertension,” although
it is unrelated to the speaker’s condition. This point should be
improved by revising the search rules after consultation withregulatory experts or using machine learning to deal with
ambiguity.
Also, since only 1 of 14 pharmacies in a single chain participatedin this study, there is a possibility that the search rules were
optimized for patients receiving prescriptions from specificmedical departments. In the experimental results, the mostfrequent ICD-10 code was “dizziness and giddiness.” As shownin Table 1, the target pharmacy frequently dispenses
prescriptions from otolaryngologists, and the results may reflect
this potential bias. Before the practical application of the system,
it is necessary to improve the search rules by considering awider range of medication histories including data from othercommunity pharmacies.
ICD-10 codes were used as normalization terms for patients’complaints regarding their symptoms because they are widely
available and understood, but MedDRA is thought to be moresuitable for extracting information on ADRs and for signal
detection. We are currently enhancing the system toaccommodate MedDRA terms.

Conclusions
In this study, we developed an automated system to extract
terms related to symptoms from the verbal complaints ofJapanese patients. As a result of an evaluation experimentcomparing automated with manual extraction, the system
performed at the level of .66 in precision, .63 in recall, and .65
for the F-measure. Although the accuracy of the system was
not satisfactory, our results suggest that it might be useful inextracting and standardizing patients’ expressions related tosymptoms from massive amounts of free text data instead of
performing those procedures manually. After improving theextraction accuracy, we expect to utilize this system to detect
the signals of ADRs from patients’ complaints in the future.

Acknowledgments
The authors wish to thank Holon Co, Ltd For agreeing to participate in this study by providing anonymized EMHD records from
September 1, 2015, to August 31, 2016. This research was supported by AMED under grant #JP16mk0101058.

Conflicts of Interest
One of the authors (TS) is an employee of Holon Co, Ltd.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.12
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et alMultimedia Appendix 1
Search rules for the program.
[XLSX File (Microsoft Excel File), 40KB - medinform_v6i3e11021_app1.xlsx ]References
1.
2.
3.
4.
5.
6.
7.
8.

9.

10.

11.
12.
13.

14.
15.

16.

17.
18.
19.
20.

21.

22.

Nebeker JR, Barach P, Samore MH. Clarifying adverse drug events: a clinician's guide to terminology, documentation, andreporting. Ann Intern Med 2004 May 18;140(10):795-801. [Medline: 15148066]
Venulet J. Progress in Drug Research. Basel: Birkhauser Verlag, Basel; 1997:233-292.
Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis ofprospective studies. JAMA 1998 Apr 15;279(15):1200-1205. [Medline: 9555760]
Budnitz DS, Lovegrove MC, Shehab N, Richards CL. Emergency hospitalizations for adverse drug events in older Americans.
N Engl J Med 2011 Nov 24;365(21):2002-2012. [doi: 10.1056/NEJMsa1103053] [Medline: 22111719]
Ernst FR, Grizzle AJ. Drug-related morbidity and mortality: updating the cost-of-illness model. J Am Pharm Assoc (Wash)
2001 Apr;41(2):192-199. [Medline: 11297331]
Rogers AS. Adverse drug events: identification and attribution. Drug Intell Clin Pharm 1987 Nov;21(11):915-920. [Medline:
3678067]
Basch E. The missing voice of patients in drug-safety reporting. N Engl J Med 2010 Mar 11;362(10):865-869 [FREE Fulltext] [doi: 10.1056/NEJMp0911494] [Medline: 20220181]
Barbara AM, Loeb M, Dolovich L, Brazil K, Russell M. Agreement between self-report and medical records on signs andsymptoms of respiratory illness. Prim Care Respir J 2012 Jun;21(2):145-152 [FREE Full text] [doi: 10.4104/pcrj.2011.00098][Medline: 22273629]
Avery AJ, Anderson C, Bond CM, Fortnum H, Gifford A, Hannaford PC, et al. Evaluation of patient reporting of adverse
drug reactions to the UK 'Yellow Card Scheme': literature review, descriptive and qualitative analyses, and questionnaire
surveys. Health Technol Assess 2011 May;15(20):1-234, iii [FREE Full text] [doi: 10.3310/hta15200] [Medline: 21545758]
Belaise C, Gatti A, Chouinard V, Chouinard G. Patient online report of selective serotonin reuptake inhibitor-induced
persistent postwithdrawal anxiety and mood disorders. Psychother Psychosom 2012;81(6):386-388 [FREE Full text] [doi:10.1159/000341178] [Medline: 22964821]
Patient-Friendly TL(V2. The MSSO. 0) URL: https://www.meddra.org/patient-friendly-term-list [accessed 2018-07-07][WebCite Cache ID 70iTr7Lyt]
Freund PR, Rowell LB, Murphy TM, Hobbs SF, Butler SH. Blockade of the pressor response to muscle ischemia by sensory
nerve block in man. Am J Physiol 1979 Oct;237(4):H433-H439. [doi: 10.1152/ajpheart.1979.237.4.H433] [Medline: 495728]
Wong A, Plasek JM, Montecalvo SP, Zhou L. Natural Language Processing and Its Implications for the Future of Medication
Safety: A Narrative Review of Recent Advances and Challenges. Pharmacotherapy 2018 Jun 09. [doi: 10.1002/phar.2151][Medline: 29884988]
Uzuner O, South BR, Shen S, DuVall SL. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J
Am Med Inform Assoc 2011;18(5):552-556 [FREE Full text] [doi: 10.1136/amiajnl-2011-000203] [Medline: 21685143]
Li Y, Salmasian H, Harpaz R, Chase H, Friedman C. Determining the reasons for medication prescriptions in the EHR
using knowledge and natural language processing. AMIA Annu Symp Proc 2011;2011:768-776 [FREE Full text] [Medline:
22195134]
Rudd RA, Aleshire N, Zibbell JE, Gladden RM. Increases in Drug and Opioid Overdose Deaths--United States, 2000-2014.
MMWR Morb Mortal Wkly Rep 2016 Jan 01;64(50-51):1378-1382 [FREE Full text] [doi: 10.15585/mmwr.mm6450a3][Medline: 26720857]
Shang N, Xu H, Rindflesch TC, Cohen T. Identifying plausible adverse drug reactions using knowledge extracted from theliterature. J Biomed Inform 2014 Dec;52:293-310 [FREE Full text] [doi: 10.1016/j.jbi.2014.07.011] [Medline: 25046831]
Shetty KD, Dalal SR. Using information mining of the medical literature to improve drug safety. J Am Med Inform Assoc
2011;18(5):668-674 [FREE Full text] [doi: 10.1136/amiajnl-2011-000096] [Medline: 21546507]
Hazell L, Shakir SAW. Under-reporting of adverse drug reactions : a systematic review. Drug Saf 2006;29(5):385-396.
[Medline: 16689555]
Botsis T, Nguyen MD, Woo EJ, Markatou M, Ball R. Text mining for the Vaccine Adverse Event Reporting System:
medical text classification using informative feature selection. J Am Med Inform Assoc 2011;18(5):631-638 [FREE Fulltext] [doi: 10.1136/amiajnl-2010-000022] [Medline: 21709163]
Xu H, Stenner SP, Doan S, Johnson KB, Waitman LR, Denny JC. MedEx: a medication information extraction system forclinical narratives. J Am Med Inform Assoc 2010;17(1):19-24 [FREE Full text] [doi: 10.1197/jamia.M3378] [Medline:
20064797]
Carrell DS, Cronkite D, Palmer RE, Saunders K, Gross DE, Masters ET, et al. Using natural language processing to identify
problem usage of prescription opioids. Int J Med Inform 2015 Dec;84(12):1057-1064. [doi: 10.1016/j.ijmedinf.2015.09.002][Medline: 26456569]

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.13
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
23.

24.

25.

26.

27.
28.

29.
30.

31.
32.
33.
34.
35.
36.
37.

Usui et al

Haerian K, Varn D, Vaidya S, Ena L, Chase HS, Friedman C. Detection of pharmacovigilance-related adverse events using
electronic health records and automated methods. Clin Pharmacol Ther 2012 Aug;92(2):228-234 [FREE Full text] [doi:10.1038/clpt.2012.54] [Medline: 22713699]
Gonzalez-Hernandez G, Sarker A, O'Connor K, Savova G. Capturing the Patient's Perspective: a Review of Advances in
Natural Language Processing of Health-Related Text. Yearb Med Inform 2017 Aug;26(1):214-227 [FREE Full text] [doi:10.15265/IY-2017-029] [Medline: 29063568]
Nikfarjam A, Sarker A, O'Connor K, Ginn R, Gonzalez G. Pharmacovigilance from social media: mining adverse drug
reaction mentions using sequence labeling with word embedding cluster features. J Am Med Inform Assoc 2015
May;22(3):671-681 [FREE Full text] [doi: 10.1093/jamia/ocu041] [Medline: 25755127]
Topaz M, Lai K, Dhopeshwarkar N, Seger DL, Sa'adon R, Goss F, et al. Clinicians' Reports in Electronic Health Records
Versus Patients' Concerns in Social Media: A Pilot Study of Adverse Drug Reactions of Aspirin and Atorvastatin. DrugSaf 2016 Mar;39(3):241-250. [doi: 10.1007/s40264-015-0381-x] [Medline: 26715498]
White RW, Wang S, Pant A, Harpaz R, Shukla P, Sun W, et al. Early identification of adverse drug reactions from searchlog data. J Biomed Inform 2016 Feb;59:42-48. [doi: 10.1016/j.jbi.2015.11.005] [Medline: 26610385]
Freifeld CC, Brownstein JS, Menone CM, Bao W, Filice R, Kass-Hout T, et al. Digital drug safety surveillance: monitoring
pharmaceutical products in twitter. Drug Saf 2014 May;37(5):343-350 [FREE Full text] [doi: 10.1007/s40264-014-0155-x][Medline: 24777653]
Denecke K, Krieck M, Otrusina L, Smrz P, Dolog P, Nejdl W, et al. How to exploit twitter for public health monitoring?
Methods Inf Med 2013;52(4):326-339. [doi: 10.3414/ME12-02-0010] [Medline: 23877537]
Aramaki E, Miura Y, Tonoike M, Ohkuma T, Mashuichi H, Ohe K. TEXT2TABLE: Medical text summarization system
based on named entity recognition and modality identification. 2009 Presented at: Workshop on BioNLP; June 4-5,2009;
Colorado p. 185-192.
Aramaki E, Yano K, Wakamiya S. MedEx/J: A One-Scan Simple and Fast NLP Tool for Japanese Clinical Texts. Stud
Health Technol Inform 2017;245:285-288. [Medline: 29295100]
Medis. Standard disease name master for ICD-10 URL: http://www2.medis.or.jp/stdcd/byomei/byomei.html [accessed
2018-07-06] [WebCite Cache ID 70iJsldRa]
MeCab. MeCab: Yet Another Part-of-Speech and Morphological Analyzer URL: http://taku910.github.io/mecab/ [accessed
2018-05-10] [WebCite Cache ID 6zJAQ4y1e]Friedman C, Hripcsak G. Evaluating natural language processors in the clinical domain. Methods Inf Med 1998
Nov;37(4-5):334-344. [Medline: 9865031]
Hripcsak G, Rothschild AS. Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc
2005;12(3):296-298 [FREE Full text] [doi: 10.1197/jamia.M1733] [Medline: 15684123]
Ministry of Health, Labour and Welfare of Japan. Report on Public Health Administration and Services FY2016 URL:
https://www.mhlw.go.jp/english/database/db-hss/rophas.html [accessed 2018-07-06] [WebCite Cache ID 70iKpMh52]
Japan Pharmaceutical Society. Report on insurance dispensing trends URL: http://www.nichiyaku.or.jp/activities/division/
faqShinchoku.html [accessed 2018-07-06] [WebCite Cache ID 70iLDq7Pi]

Abbreviations
ADE: adverse drug event
ADR: adverse drug reaction
AMED: Japan Agency for Medical Research and Development
CRF: conditional random fields
EHR: electronic health record
EMHD: electronic medication history data
FAERS: US Food and Drug Administration Adverse Event Reporting System
ICD-10: International Classification of Diseases, Tenth Revision
MedDRA: Medical Dictionary for Regulatory Activities
NLP: natural language processing

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.14
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Usui et al

Edited by G Eysenbach; submitted 12.05.18; peer-reviewed by MA Mayer, KNB Nor Aripin; comments to author 13.06.18; revised
version received 07.08.18; accepted 25.08.18; published 27.09.18.
Please cite as:
Usui M, Aramaki E, Iwao T, Wakamiya S, Sakamoto T, Mochizuki M
Extraction and Standardization of Patient Complaints from Electronic Medication Histories for Pharmacovigilance: Natural Language
Processing Analysis in Japanese
JMIR Med Inform 2018;6(3):e11021
URL: http://medinform.jmir.org/2018/3/e11021/
doi:10.2196/11021
PMID:30262450©Misa Usui, Eiji Aramaki, Tomohide Iwao, Shoko Wakamiya, Tohru Sakamoto, Mayumi Mochizuki. Originally published in
JMIR Medical Informatics (http://medinform.jmir.org), 27.09.2018. This is an open-access article distributed under the terms of
the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly
cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as thiscopyright and license information must be included.

http://medinform.jmir.org/2018/3/e11021/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e11021 | p.15
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Original Paper

Uncovering a Role for Electronic Personal Health Records inReducing Disparities in Sexually Transmitted Infection Rates
Among Students at a Predominantly African American University:
Mixed-Methods Study
Kevon-Mark Jackman1, DrPH, MPH; Stefan David Baral1, MD, MPH; Lisa Hightow-Weidman2, MD, MPH; Tonia
Poteat1, PhD, MPH
1Center for Public Health and Human Rights, Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, UnitedStates2

Institute for Global Health and Infectious Diseases, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States

Corresponding Author:
Kevon-Mark Jackman, DrPH, MPH
Center for Public Health and Human Rights
Department of Epidemiology
Johns Hopkins Bloomberg School of Public Health
E7143
615 N Wolfe Street
Baltimore, MD, 21205
United States
Phone: 1 352 871 1286
Email: kjackma2@jhmi.edu

Abstract
Background: Black youth continue to bear an overwhelming proportion of the United States sexually transmitted infection(STI) burden, including HIV. Several studies on web-based and mobile health (mHealth) STI interventions have focused on
characterizing strategies to improve HIV-related prevention and treatment interventions, risk communication, and stigma among
men who have sex with men (MSM), people who use substances, and adolescent populations. The Electronic Sexual Health
Information Notification and Education (eSHINE) Study was an exploratory mixed-methods study among students at a historically
black university exploring perceptions on facilitating STI testing conversations with partners using electronic personal healthrecords (PHRs).
Objective: The purpose of this paper is to use eSHINE Study results to describe perceived impacts of electronic PHRs on
facilitating STI testing discussions between sexual partners.
Methods: Semistructured focus groups and individual in-depth interviews were conducted on a heterogeneous sample of students(n=35) between May and July 2014. Qualitative phase findings guided development of an online survey instrument for quantitative
phase data collection. Online surveys were conducted using a convenience sample of students (n=354) between January and May
2015. Online survey items collected demographic information, sexual behaviors, beliefs and practices surrounding STI testing
communication between partners, and beliefs about the impact of electronic PHR access on facilitating these discussions withpartners. Chi-square analysis was performed to assess gender differences across quantitative measures. A Wilcoxon signed rank
sum test was used to test the null hypothesis that electronic PHRs are believed to have no effect on the timing of dyadic STI
health communication.
Results: Participants described multiple individual and dyadic-level factors that inhibit initiating discussions about STI testing
and test results with partners. Electronic PHRs were believed to improve ability to initiate conversations and confidence in STI
screening information shared by partners. Among online survey participants, men were more likely to believe electronic PHRs
make it easier to facilitate STI talks with potential partners (59.9% vs 51.9%; χ2=3.93, P=.05). The Wilcoxon signed-rank test
results indicate significant increases in perceived discussion timing before sex with electronic PHR access (61.0% vs 40.4%;
P<.001).

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.16
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Conclusions: Findings suggest that electronic PHR access in STI screening settings among similar populations of Black youth
may improve both motivation and personal agency for initiating dyadic STI health communication. Results from this study will
likely inform novel interventions that use access to electronic PHRs to stimulate important health-related discussions between
sexual partners. Moving forward requires studying strategies for implementing interventions that leverage electronic PHRs tocreate new sexual health communication channels with providers, peers, and family among black youth.
(JMIR Med Inform 2018;6(3):e41) doi:10.2196/medinform.9174
KEYWORDS
PHR; STI; HIV; mHealth; intervention; prevention; young people; black; health disparities

Introduction
The High Burden of Sexually Transmitted Infections
Among Black Youth and Dyadic Sexually Transmitted
Infection Testing Talks
Young black people in the United States (US) are largely
overrepresented in cases of sexually transmitted infections(STIs), including HIV. While black people constituteapproximately 15.4% of US youth ages 15 to 24 years, theyaccounted for 29.9% of chlamydia cases, 47.7% of gonorrhea
cases, 43.3% of early syphilis, and 54.7% of HIV cases
diagnosed in this age group in 2016 [1-3]. Getting young people
to talk about STI testing, including HIV, with partners prior tosex is a critical challenge to disease prevention and control [4].
These preventative conversations support testing, disease status
disclosure, condom use, and the use of medicines to preventand treat STIs [5]. The “Get Yourself Tested” campaign
launched nationally in 2009 in effort to reduce STIs in young
people by targeting four key behaviors: STI testing, HIV testing,talking to partners about testing, and talking to providers about
testing. Formative research used to inform the campaign design
found that STIs remain highly stigmatized and as a result, few
young people discuss testing with partners prior to sex [4,6].
In young people, conversations on STI testing are more likely
to be facilitated by those that routinely screen for STIs [7]. In
studies of students at historically black colleges and universities,
women reported higher screening rates and asked partners about
HIV status more than their male counterparts [8,9]. Additionally,
individuals are more likely to discuss STI status and risk
behaviors with a main partner compared to a casual partner [10].
Expected partner reactions of suspicion, accusation, or insult(from inadvertently implying partner distrust) are sometimes
barriers to discussing STI testing [4]. Barriers that inhibit partner
communication regarding STI risk restricts the ability to makewell-informed decisions about the level of risk assumed andemployable methods to prevent disease transmission [11,12].

Digital Prevention Tools
The US National HIV Strategy calls for federal agencies toencourage the development and implementation of highly
accessible digital tools to educate and inform the American
people with scientifically accurate information on disease risk,
prevention, transmission, and treatment [13]. Several studies
on Web-based and mobile health (mHealth) HIV/STI
interventions have been conducted and are currently ongoingthat focus on improving outcomes related to testing, risk
communication, reducing stigma, increasing condom use, andhttp://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

improving adherence to biomedical prevention and treatment
among men who have sex with men (MSM), substance using,and adolescent populations [14-19]. Although mHealthintervention studies focused on improving partner
communication on STI testing are promising, most studies do
not account for the role of remote access to STI screening
records in dyadic STI health communication. Electronic personal
health records (PHRs) or patient portal services (eg, Epic
MyChart, Cerner Patient Portal, Kaiser Permanente My HealthManager, Quest Diagnostics Care360) provide patients withremote access to their laboratory test results, including STI
results. Electronic PHRs differ from most mHealth platforms
in STI intervention studies since electronic PHRs containprotected health information managed by covered entities,
therefore, federal regulations require electronic PHR vendors
to adhere to data security and integrity measures outlined by
the Secretary of the Department of Health and Human Services[20]. Nevertheless, electronic PHRs can provide several services
in addition to medical record access, like educational resources
and tools for communicating with healthcare providers.

Study Purpose
The Electronic Sexual Health Information Notification and
Education (eSHINE) Study was a mixed-methods study among
students 18 to 25 years old attending a historically black
university. The project was completed using a dissertation
research funding grant from the Agency for Healthcare Research
and Quality, and explores perceptions of using electronic PHRs
to share electronic STI screening information between sexual
partners. The current study is an analysis of eSHINE Study
qualitative and quantitative data to explore attitudes andpractices surrounding STI testing talks between partners, barriers
to talking about STI testing with partners, and perceived impactsof electronic PHRs on risk discussion facilitation. Thus, thepurpose of our study is to provide a rich contextual
understanding of how diffusing electronic PHR access in ourstudy population may impact health-related communication
between sexual partners.

Methods
Study Design
Exploratory mixed methods are a two-phase sequential study
design that is particularly useful for exploring new researchquestions [21]. It calls for an initial qualitative exploration phase(ie, focus-groups and individual interviews), an intermediatesurvey development phase, and a second quantitative researchphase (ie, an online survey) [22]. A constructivist approach of
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.17
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
incorporating multiple theoretical frameworks structured ourstudy protocol. The Integrative Model of Behavioral Prediction(IMBP), often used for designing health messages, posits thatbehavioral intentions are a function of attitude, normative, andself-efficacy beliefs [23]. IMBP also considers the impact ofdemographic variables, culture, and environmental constraints
on behavior. If electronic PHR–facilitated STI talks is a novel
concept to the study population, constructs of the Diffusion of
Innovation Theory (DOI), such as innovation attributes andcommunication channels, were incorporated into our exploration[24]. Finally, the Disclosure Processes Model provide
considerations for individuals with a history of STI diagnosis
or exposure; here, disclosing risk is a function of antecedentgoals (ie, approach goals vs avoidance goals) [25].

Recruitment
Eligible study participants were students, ages 18 to 25 years,
enrolled at a southern historically black college and university
at the time of study. Qualitative phase participants were recruited
between May and July 2014 and quantitative phase participants
between January and May 2015. Recruitment flyers were posted
on campus along with multiple announcements sent throughthe university’s student, faculty, and staff email list. Targeted
recruitment efforts were conducted in collaboration with
University Health Services, the Student Counseling Center, theOffice for Residence Life to post study materials and conducttabling events. Study flyers were posted to a university affiliatedFacebook page for lesbian, gay, bisexual, transgendered, andquestioning students. Participants were also recruited at student
organizational meetings, including theatre, peer educator, andfootball. Study enrollment included providing eligible students
with a detailed description of the project, participation
requirements, and terms for receiving incentives. To complete
enrollment, prospective participants signed an informed consent
form in person or online using Adobe Echosign. Qualitativeand quantitative phase research protocols, including focus-group,interview guides and online survey instruments were separately
reviewed and approved by the university’s Institutional Review
Board.
Qualitative phase participants received US $25 cash for eachsession (limited to one focus group or one interview per
participant). Quantitative phase participants received US $20
cash for completing the online survey; and if eligible, qualitative
phase participants could participate in the quantitative phase.
Online survey participants were not asked to provide their nameor email address on the survey. To receive incentives,
participants provided a unique code generated at survey
completion. Participants were additionally entered to win prizes
in one of three raffle drawings (eg, textbook vouchers, US $25
up to US $100 gift cards, Samsung Galaxy tablets). eSHINE
Study data collection and analyses was conducted by KJ as partof his dissertation research, with guidance from project mentors
and dissertation committee members. Prior to the study, KJ
completed training in qualitative and quantitative researchmethods, including academic coursework and work as an STD
prevention and control program disease intervention specialist.

Jackman et al

Qualitative Methods
A total of 35 students participated in the qualitative researchphase (19 male; 16 female). Audio-recorded focus group andindividual interview sessions were conducted by KJ insideprivate conference rooms located on the university’s campus.
In May 2014, 33 students participated in one of three separate
focus group sessions (n=6; n=10; n=17). Semistructured sessions
averaging 70 minutes in length were divided into three
discussion sections: (1) electronic PHR perceptions, (2)experiences and perceptions related to dyadic STI healthcommunication, and (3) perceptions related to using electronic
PHRs in dyadic STI health communication. At the end of eachfocus group, participants were invited to schedule an in-depth
individual interview session. Focus group and interview
recordings were played back after sessions by KJ to constructfield notes and inform any modifications to main questions insubsequent sessions.

Semistructured interviews were important to explore in depth
the statements made by participants during focus groups in asetting isolated from peers. On average, interviews lasted 45
minutes. An oral questionnaire was administered during
interviews to collect demographic information, orientation, andsexual risk behavior, such as number of recent sex partners,
condom usage, and sex under the influence of alcohol or drugs
practices. Considering the sensitivity of sexual topics and tosupport higher levels of comfort among participants, thisinformation was not collected during focus groups. Sixteen ofeighteen individual in-depth interviewees were recruited froma focus group session. Transcripts and field notes were uploaded
to ATLAS.ti to facilitate the qualitative analysis [26]. The
analysis included reading through transcripts and field notes
while identifying useful quotes or sentences, creating memos,
coding segments of information, assigning labels to codes, andthe grouping of codes into broad themes [21].

Survey Development
A Qualtrics online survey was developed during the intermediatesurvey development phase between July and December 2014[27]. Qualitative codes were operationalized into IMBP
behavioral construct variables, such as attitudinal, self-efficacy,
and intentional beliefs within broad emergent themes. The
operationalization process included using individual codes
within themes as variables as well as using specific quotes fromparticipants within items on the survey [21]. Behavioral
construct variables, for example, attitudes on the importance ofdiscussing STI testing with partners, were measured using
7-point Likert scale items, scored –3 to 3 [28]. Data was alsocollected on several demographic and sexual behavior variables,such as gender, screening practices, and risky sex behaviors.
The survey was piloted with eight students and revisions madebased upon participant feedback and researcher observations.
The final survey had 116 items and a completion time of
approximately 30-45 minutes.

Quantitative Methods
To access the online survey, a secured and unique Web-link
was sent to the enrolled participant’s student email account

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.18
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
using the university’s email database. Between January and
May 2015, 1093 participants registered for the online survey
and were emailed secured survey links; 501 surveys werestarted, 380 completed, and 354 completed without missing
data.
Survey data were uploaded into STATA 14 for statisticalanalyses [29]. Chi-squared analyses were conducted by genderto describe distributions of demographic information, risk
behaviors, STI testing talk attitudes and practices, andperceptions of electronic PHR–facilitated discussions by gender.
Effect size (Cohen d) was calculated for variables demonstrating
significant differences by gender related to electronic PHR
impacts on discussions. A Wilcoxon signed rank sum test was
performed to determine differences in perceived discussion
timing with electronic PHR access compared to without
electronic PHR access. To test the null hypothesis that electronic
PHRs are believed to have no effect on the timing of STI
conversations, we compared responses on two survey items:(1) “When do discussions about STD testing typically occur
between you and your partner(s)?” and (2) “When would
discussions about STD testing likely occur if you and yourpartner(s) had electronic PHRs?” Reponses were ranked from
1 to 4; (1) never (2) after sex (3) inconsistently before or aftersex, and (4) before sex.

Results
Qualitative Phase Results
Qualitative phase participants were heterogeneous in academicclassification, degree major, sexual behaviors, sexual orientation,
including for example, student athletes, peer educators, andmembers of Greek lettered social organizations. Table 1 presentsinterview and focus group quotations from our qualitative phasestudy. For this study, quotations were categorized under two
major themes: attitudes and practices surrounding dyadic STI
health communication and expected impacts of electronic PHRs
on dyadic STI health communication. Table 1 also indicates
online survey measures derived from qualitative codes.

Sexually Transmitted Infection Testing Talks:
Qualitative Attitudes and Practices
STI testing talks or risk discussion events were described asverbal exchanges between sexual partners regarding STI testingor status. Soliciting information related to STI risk from partners
was described as demonstrating personal responsibility and an
unalienable right to self-preservation:
You have the right to know.
I feel like if you’re having sex with me you have theright to know my STD history. If I’m having sex withyou, I need to know everything too. I owe that to you,
you owe that to me.
Conversations vary in both timing and depth. Several multi-levelfactors, such as self-efficacy, partner-type, and intoxication,
can impact when and how discussions occur (Table 1). Whentalks occur, they are either distal, proximal, or after the sexual

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

Jackman et al
encounter. Distal discussions happen in a period whereindividuals are “getting to know each other.” While proximal
discussions occur when participants are “in the moment.”
Discussions can range in depth from simply asking, “are you
good down there?” to requiring current screening informationand information regarding condom use practices with previous
partners.
Valuation for risk discussions varied between participants. For
most, STI talks are very important to always occur, especiallyinterviewees disclosing a history of STI infection.
It’s very important to have that conversation.
Some participants saw no purpose in talking about STI testing
when using condoms. Valuation also varied based on dyadic
characteristics such as, partner type.
If I know that I am going to be in somethingcommitted then I want to know your history. But if
you are just a casual partner then I don’t really care,because I am going to protect myself. I wouldn’t havethe conversation with someone I’m just casually
having sex with.
Participants described sexual partner dynamics ranging fromsolely pleasure-seeking sex and noncommittal partnerships tosocio or emotional interdependent and committed relationships.
To simplify and operationalize, partner types were classified
as: (1) main partners, defined as partnerships intended to be
exclusive relationships; (2) casual partners, defined as recurring
partnerships not intended to be exclusive relationships; and (3)hook-up partners, defined as one-time partnerships. Keeping in
mind that dyad characteristics vary within partner-typeclassifications (for example, casual partners may be long-timefriends or recent acquaintances, and others).
Finally, valuation appears to also be determined by the extent
to which individuals are aware of the importance of discussing
STI testing with partners.
I have never discussed STDs with any of my sexual
partners. I’m young, so it never really came up. I
usually just say, what’s the number of people you
have had sex with and if I feel comfortable with thenumber then, okay.

Barriers to Talking About Sexually Transmitted
Infection Testing With Partners: Attitudes andPersonal Agency
Self-efficacy contributes to whether STI testing talks are
initiated. While some participants described being very
comfortable with initiating conversations, many generally
describe it as an “[i]t’s an awkward conversation.” The impactof these beliefs as an inhibitor is based on the level of disruption
an individual believes the conversation will cause to a potential
sexual encounter or relationship. Low self-efficacy to facilitaterisk discussions was mostly described in the context of proximal
discussions, where consequences of these events may “ruin the
mood.” Lack of self-efficacy to initiate STI testing talks without
ruining the mood presents as a primary barrier to some.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.19
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Table 1. Focus-group and individual in-depth interview quotations and online survey measures derived from qualitative codes. PHR: personal healthrecord; STI: sexually transmitted infection.
Themes and online survey measure

Quotations

Attitudes and practices surrounding dyadic STI health communication
Timing of dyadic STI health communication••

It usually occurs before anything else. After I get to know the person, it
comes up in the conversation because this person is a potential partner. I
always ask to be on the safe side.
Before I have sex, we don’t have the conversation. It slips my mind until
after we have sex.

•

It is very important to have that conversation, especially if you are going
to be dealing with that person sexually, “Have you been tested?” “Are you
going to get tested?” “When was the last time you were tested?

Valuation for dyadic STI communication when using condoms; •communications barrier: condom use

If I know that I am going to be in something committed, then I want to know
your history. But if you are just a casual partner then I don’t really care,because I am going to protect myself.

Self-efficacy to initiate dyadic STI health communication•

It doesn’t make me uncomfortable, I am straightforward. If I feel that weare about to get serious or have any sexual encounters, I simply ask “whenwas the last time you got tested?” If it’s too long, I tell them where to go
for testing.

Communication barrier: precontemplation••

When I was fresh out of high school, out of my parent’s house, I was[sleeping with] 6 or 7 guys at the same time. I was young, so I did not think
to ask, have you gotten tested, how many people have you been with?
I have never discussed STDs with any of my sexual partners.

Communication barrier: awkward••

It kills the mood.
(It can be awkward) when you have known the person.

Communication barrier: people lie•

People lie. One of the big lies is “I’ve been tested” or “I don’t have anything.” Especially when you’re in the moment, it happens all the time.

Valuation for dyadic STI health communication

Expected impacts of electronic PHRs on Dyadic STI Health Communication
More confidence in STI testing information shared by a partner •It’s another way of verifying the truth and showing that they did get tested
or if we need to get tested—we can go [to a test site] together.

•

Ultimately the app would make it much easier to have these conversations
with someone you are going to have sex with—whether it’s casual or longterm.
I can just show my partners casually when I got tested and my results. It
will ease the tension and make it more comfortable, especially if I am
willing to share that information with you.

Easier for potential partners to talk about STI testing; easiercheck-in talks with partners on STI testing•

Impact on frequency of STI talks; earlier STI talks (proximity
to potential encounter)••

Intentional beliefs to only use electronic PHRs when distrusting •of partners

I wouldn’t have a need for it, but then again if I do want to, you shouldn’t
be offended, because I’m just trying to be safe. So, if I do bring it up, don’t
be upset because it’s good health.

•If I tell you something and you don’t believe it, we shouldn’t be having sex
in the first place. If I tell you something, that’s what you should believe. If
I am lying, then strap up [use a condom].
It’s tricky, it’s one thing to ask someone something, but then to tell themto verify it, it messes up the trust. Unless if they are very comfortable.

Soliciting a partner’s electronic record will be awkward•

Self-efficacy for sharing a positive STI electronic PHR; preferred method to share STI positive status

Suspicious of partners unwilling to share electronic PHR
http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

If the norm was for people to have the app at hand, then more people would
ask to see results. Now, it’s not that realistic, because people can easily
say, I don’t have it with me, it’s on paper.
If the app is popular, then I’m asking everybody.

••

People who are negative would gladly show their results. People who arepositive, it would be harder for them.
I don’t see anyone showing a partner the app unless if they are clean.

•

Red flag.
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.20
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Some participants described talking with partners about STI
testing as an exercise in futility because of limitations in theability to verify shared information.
People lie. One of the big lies is “I’ve been tested”
or “I don’t have anything”. Especially when you’rein the moment, it happens all the time.
Allaying a partner’s STI transmission concerns can takeprecedence over communicating accurate information about
STI screening and risk. Several approaches are employed tomitigate risk associated with receiving inaccurate information.
For example, some participants reported choosing to avoid risk
discussions all together and use condoms. Conversely, someparticipants said that they engaged in couples testing or requirepotential partners show verification of STI test results prior tosex.
Any boy I ever ask that question to, I make sure I see
papers. Paper says clearly negative or positive.
Though some participants commented that it is unlikely for“papers” to be readily available to validate STI screening
information.
Other barriers exist to starting dyadic talks on STI testing.
Substance use prior to sex was considered to inhibit ability forfacilitating risk discussions with partners.
If someone is drunk, then it’s not going to be
discussed.
Self-efficacy to initiate risk discussions may also be diminishedwhen dyads have a previously established intimate relationshipor a friendship. Difficulty discussing STI testing “when you
have known the person” was described by participants as
implying distrust.

Impact of Electronic Personal Health Records on
Sexually Transmitted Infection Testing Talks: Beliefs
and Expectations
Information verification was considered the foremost benefit
of incorporating electronic PHRs into risk discussions between
sexual partners.
It’s another way of verifying the truth and showingthat they did get tested or if we need to get tested—
we can go (to a test site) together.
Participants used terms such as, “truth-detector,” “proof,” and“confirmation,” in referring to electronic PHR use with partners.
Participants also described limitations of electronic PHRs indetermining a potential partner’s real-time infection status.
There is no way to say you are clean today, but you
can say you were clean that day.
Together, electronic PHRs are believed to be a compatible
innovation for adding assurance to STI talks.
I think this will be something good for the gay
community. The gay community is big on electronic
dating and meeting people online, Grindr and Jack'd
and all that. I feel that it would be really good forthat.

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

Jackman et al
Gaining electronic PHR access was believed to ease the ability
to facilitate conversations on STI testing.
I can just show my partners casually when I got tested
and my results. It will ease the tension and make it
more comfortable, especially if I am willing to sharethat information with you.
Another participant explained:
Ultimately the app would make it much easier to havethese conversations with someone you are going tohave sex with—whether it’s casual or long term.
Participants added that electronic PHRs could make it easier tohave “check-in” conversations with partners who have priorestablished intimate relationships or friendships. Someparticipants maintained that practices will be determined bypartner-type.
If I have a one-night stand, I will use protection. I
would not want to ask that question.
However, for most, the idea of an easier STI talk eliminates
partner-type related factors as a barrier.
If the app is popular then I’m asking everybody.
Improvements to personal agency was not anticipated acrosssome constraining conditions, such as intoxication or being
infected with an STI.
If you’re drunk, I don’t think people would use it,
because you really wouldn’t be thinking about that,your mind is somewhere else.
Similarly, it was expressed that individuals with electronic
records positive for STI infection may employ strategies toavoid talking with partners about STI screening records.
People who are negative would gladly show theirresults. People who are positive, it would be harderfor them.
Nevertheless, some participants believed that incorporating
educational resources within electronic PHR products mightprove useful in explaining positive test results and prevention.
While electronic PHRs were compatible with most participants
as a potential tool for facilitating STI talks, some participantssuggest electronic PHR solicitation as intrusive or implying
distrust.
It’s tricky, it’s one thing to ask someone something,but then to tell them to verify it, it messes up the trust.
Unless if they are very comfortable.
Additionally, partners unwilling to share electronic screening
records are anticipated to raise a “red flag” regarding future
sexual decisions and relationship progression. Overall, electronic
PHR access was anticipated to have the population-level impact
of increasing discussions on STI testing.
If the norm was for people to have the app at hand,then more people would ask to see results. Now, it’s
not that realistic, because people can easily say, I
don’t have it with me; it’s on paper.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.21
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Quantitative Phase Results
Table 2 shows demographic information and sexual risk
behaviors among online survey participants (n=354). The sample
consisted of 167 male and 187 female participants. Forty outof 354 participants (11.3%) reported no history of sexualintercourse and 44.3% reported 2-5 partners. Approximately
43.2% (153/354) reported STI testing seven months prior to thestudy; 80 out of 354 participants with sexual exposure reported
no history of screening (22.1%). Almost half (172/354) ofparticipants reported sex under the influence of alcohol or drugs
and 134 out of 354 participants reported recent sex without
discussing STI testing (Table 2). Sixty-four percent ofparticipants (38/59) with a history of STI diagnosis reported
chlamydia infection. Additionally, five participants reporting
gonorrhea, two participants reporting HSV-2, and twoparticipants reporting human papilloma virus also reported
chlamydia infections, there was no overlap between any other
STIs (not shown).

Sexually Transmitted Infection Testing Talks:
Quantitative Attitudes and Practices
Table 3 presents behavioral perceptions on risk discussion
practices among online survey participants. Conversations
consistently occur before sex for 143 out of 354 participants(40.4%) and inconsistently occur before or after a sexual
encounter for 41.0% (145/354). Some participants (51/354)
reported never discussing STI testing with partners. Most
participants (312/354) value discussing STI testing with partners,
however, valuation was slightly greater among women (χ2=3.79;
P=.05). In fact, 71.7% (254/354) believed it is extremely
important or very important irrespective of condom-use,
however over two-fifths (158/354) reported recent sexual
encounters where risk discussions were skipped due tocondom-use. Furthermore, less than half (169/354) believe that
it is easy to talk with partners about STI testing and 28.2%(100/354) reported skipping a recent discussion due toawkwardness. One-third of participants (118/354) reported
recently skipping risk discussions due to the possibility ofreceiving inaccurate information. Many participants (112/354)
also reported discussion omission because it never came tomind.

Impact of Electronic Personal Health Records on
Discussion Timing
Out of 354 online survey participants, 184 (60.0%) believedthat electronic PHR access will lead sexual partners to start

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

Jackman et alconversations on STI testing earlier in the relationship (Table4). The Wilcoxon signed-rank test results shown in Figure 1
indicate significant differences between perceived discussion
timing with and without electronic PHRs (P<.001). Discussions
occurring before sex increased 20.6% with electronic PHR
access; while omitting talks and inconsistent timing bothdecreased by 10% (Figure 1).

Impact of Electronic Personal Health Records on
Personal Agency, Information Assurance, andAssessing Partner Risk
Table 4 presents bivariate relationships between perceptions on
incorporating electronic PHRs into risk discussion events andgender. Almost two-thirds (225/354) of participants felt thatelectronic PHRs would help to improve communication between
partners about STI prevention. Similarly, most participants(235/354) believed electronic PHRs would increase theirconfidence in STI testing information shared by a partner.
Electronic PHRs are believed to make it easier to facilitate risk
discussions between new potential partners (197/354) andongoing sexual partners (195/354). Nearly a quarter ofparticipants (85/354) believed soliciting a partner’s electronic
record would be awkward. Furthermore, two-fifths (154/354)
believed it would be difficult to share a positive result with apartner. In fact, more participants (158/354) indicated aconversation without electronic PHRs as the preferred method
for STI disclosure compared to using electronic PHRs (81/354).
The most commonly held belief was that participants will be
suspicious of potential partners unwilling to share electronic
PHRs (268/354).

Electronic Personal Health Record–Facilitated Sexually
Transmitted Infection Talks: Beliefs by Gender
Male participants were more likely to believe that electronic
PHR access would make it easier for sexual partners to discuss
STI testing compared to female participants (59.9% vs. 51.9%;
χ2=3.93; P=.05). However, the effect size by gender was small(d=0.27). Gender differences between beliefs that electronic
PHR facilitated conservations would be awkward andself-efficacy beliefs for sharing electronic STI records withpositive results had larger effect sizes. A smaller proportion ofmales (18.0% vs 29.4%) consider electronic PHR–facilitateddiscussions to be potentially awkward and were additionallymore confident in their ability to share a STI-positive electronic
result compared to female participants [(χ2=10.85; P=.001;
d=1.04) and (χ2=6.48, P=.01; d= 0.80)].

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.22
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Table 2. Demographic information and sexual risk behaviors among eSHINE Study online survey participants (n=354). IQR: interquartile range; STI:
sexually transmitted infection.
Variables

Total, n (%)

Men, n (%)

Women, n (%) Chi-square

Age
Median age (IQR)

20 (19-22)

20 (19-22)

P value

—

—

23.64

<.001

267.15

<.001

18.88

<.001

—

—

21.14

<.001

20 (19-22)

Academic classification
Freshman89 (25.1)

57 (34.1)

32 (17.1)

Sophomore

82 (23.1)

42 (25.1)

40 (21.4)

Junior87 (25.6)

37 (22.2)

50 (26.7)

Senior88 (24.9)

31 (18.6)

57 (30.5)

Graduate student

8 (2.3)0 (0.0)8 (4.3)Sexual preference by gender
Men only

172 48.6)

10 (6.0)162 (86.6)

Women only

156 (44.1)

149 (89.2)

7 (3.8)Men and women

26 (7.3)

8 (4.8)18 (9.6)

Reported sex partners (in 12 months prior)
No partners in 12 months prior to study or no history of sexual intercourse 56 (15.8)

34 (20.4)

22 (11.8)

1116 (32.8)

45 (26.9)

71 (38.0)

2

79 (22.3)

27 (16.2)

52 (27.8)

3-5

78 (22.0)47 (28.1)

31 (16.6)

6+25 (7.1)

14 (8.4)

11 (5.9)

Reported partner-typesa
Main partner(s)213 (60.2)

84 (50.3)

129 (69.0)

Casual partner(s)153 (43.2)

77 (46.1)

76 (40.6)

Hook-up partner(s)72 (20.3)

47 (28.1)

25 (13.4)

STI screening history< 7 months

153 (43.2)

53 (31.7)

100 (53.5)≥ 7 months

81 (22.9)

39 (23.3)

42 (22.5)

Never tested

80 (22.6)

51 (30.5)

29 (15.5)

No history of sexual intercourse

40 (11.3)

24 (14.4)

16 (8.6)

History of STI diagnosis

59 (16.7)

14 (8.4)

45 (24.1)

15.62

<.001

Concurrent sexual partners

68 (19.2)

38 (22.8)30 (16.0)

2.56

.11

Sex under the influence of drugs or alcohol

172 (48.6)

65 (38.9)

107 (57.2)

11.82

.001

Condom-less sex with a casual partner

106 (30.8)

44 (26.3)

65 (34.8)

2.92

.09

Condom-less sex with a hook-up/one-time partner

26 (7.3)

12 (7.2)

14 (7.5)

0.01

.91

Met sex partners using social websites or applications

56 (15.8)

39 (23.4)

17 (9.1)

13.47

<.001

Sex without discussing STI testing

134 (37.8)

62 (46.3)

72 (38.5)

0.07

.79

STI diagnosis history and risky behaviors in 12 months prior to studyaPartner type categories reported by participants are not mutually exclusive.

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.23
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Table 3. Behavioral attitudes and practices related to dyadic conversations on sexually transmitted infection (STI) testing among eSHINE Study online
survey participants, bivariate analyses by gender (n=354).
Variables

Total, n (%)

Men, n (%)

Women, n (%) Chi-square

Valuation belief for talking about STIs with partners (n=315)
Very important/important

312 (88.1)

137 (82.0)

175 (93.6)

Very unimportant/unimportant

3 (0.8)3 (1.8)0 (0.0)

Valuation belief for talking about STIs with partners when using condoms(n=263)
Very important/important

254 (71.7)

105 (62.9)

149 (79.7)

Very unimportant/unimportant

9 (2.5)6 (3.6)3 (1.6)

Self-efficacy belief to initiate risk discussions (n=213)
Very easy/easy

169 (47.7)

76 (45.5)

93 (49.7)

Very difficult/difficult

44 (12.4)

20 (12.0)

24 (12.8)

Intentional belief on likelihood to solicit STI screening from a partner(n=254)
Very likely/likely

226 (63.8)

93 (55.7)

133 (71.1)

Very unlikely/unlikely

28 (7.9)

15 (9.0)

13 (6.9)

Timing of STI testing talks with partners discussion (n=354)

P value

3.79

.05

2.29

.13

0.00

.95

1.57

.21

2.77

.43

Before sex

143 (40.4)

68 (40.7)

75 (40.1)

Sometimes before sex and sometimes after sex

145 (41.0)

64 (38.3)

81 (43.3)

After sex

15 (4.2)

6 (3.6)9 (4.8)Never

51 (14.4)

29 (17.4)

22 (11.8)

Condoms were being used

158 (44.6)

86 (51.5)

72 (38.5)

0.00

.97

The topic would make things awkward

100 (28.2)

47 (28.1)

53 (28.3)

0.14

.71

People can lie about it regardless

118 (33.3)

54 (32.3)

64 (34.2)

6.03

.01

The topic never came to mind

112 (31.6)

58 (34.7)

54 (28.9)

1.40

.24

Reasons for omitting dyadic STI health communication for sexual encounters in 12 months prior to study (n=354)

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.24
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Table 4. Perceptions on incorporating personal health records (PHRs) into risk discussion events among eSHINE Study online survey participants(n=354). STI: sexually transmitted infection.
Variables

Total, n (%)

Men, n (%)

Women, n (%) Chi-square

Semantic belief on the effect of electronic STI record access on communi-

P value

1.74

.19

0.19

.66

3.93

.05

0.53

.46

10.85

.001

6.48

.01

0.70

.40

24.14

<.001

1.49

.22

2.39

.49

cation on STIs between partners (n=227)a
Very helpful/helpful

225 (63.6)

105 (62.9)

120 (64.2)

Very harmful/harmful

2 (0.6)0 (0.0)2 (1.2)

Semantic belief on the effect of electronic STI record access on confidence
in STI testing information shared by a partner (n=238)a
Very helpful/helpful

235 (66.4)

108 (64.7)

127 (67.9)

Very harmful/harmful

3 (0.9)1 (0.6)2 (1.1)

Belief that electronic PHRs make it easier for potential partners to talkabout STI testing (n=213)

aStrongly agree/agree

197 (55 .6)

100 (59.9)

97 (51.9)

Strongly disagree/disagree

16 (4.5)

4 (2.4)

12 (6.4)

Belief that electronic PHRs make it easier to check-in with partners on
STI testing and prevention (n=206)a
Strongly agree/agree

195 (55.1)

93 (55.7)

102 (54.5)

Strongly disagree/disagree

11 (3.1)

4 (2.4)7 (3.7)

Belief that soliciting a partner’s electronic record will be awkward (n=171)a
Strongly agree/agree

85 (24.0)30 (18.0)

55 (29.4)

Strongly disagree/disagree

86 (24.3)

52 (31.1)

34 (18.2)

Self-efficacy belief for sharing a positive electronic STI record with apartner (n=213)a
Very easy/easy

59 (16.7)

36 (21.6)

23 (12.3)

Very difficult/difficult

154 (43.5)

64 (38.3)

90 (48.1)

Belief that partners using electronic PHRs will start talking about STI
prevention earlier in a relationship (n=200)a
Strongly agree/agree

184 (52.0)

89 (53.3)

95 (50.8)

Strongly disagree/disagree

16 (4.5)

6 (3.6)

10 (5.3)

Intentional beliefs to only use electronic PHRs when distrusting of partners(n=232)Strongly agree/agree

93 (26.3)

62 (37.1)

31 (16.6)

Strongly disagree/disagree

139 (39.3)

47 (28.1)

92 (49.2)

Attitudinal belief of being suspicious of partners unwilling to share electronic PHR (n=270)a
Strongly agree/agree

268 (75.7)

117 (68.9)

153 (81.8)

Strongly disagree/disagree

2 (0.6)0 (0.0)2 (1.1)

Preferred method for sharing a positive infection status (n=354)aUsing an electronic PHR

81 (22.9)42 (25.2)

39 (20.9)

A conversation without electronic PHRs

158 (44.6)

68 (40.7)

90 (48.1)

Avoid sharing infection status

16 (4.5)9 (5.4)7 (3.7)

No preference

99 (28.0)

48 (28.7)

51 (27.3)

Scores between –1 and 1 for belief variables are not reported.

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.25
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Jackman et al

Figure 1. Perceived sexually transmitted infection communication timing with sexual partners with and without electronic personal health record (PHR)
access among eSHINE Study online survey participants (n=354). The Wilcoxon signed-rank test indicates significant increases in perceived discussion
timing before sex with electronic PHR access (P< .001).

Discussion
Principal Results
Together, qualitative and quantitative findings offer several
considerations for the potential role of electronic PHRs in
facilitating STI health communication between partners. STI
health communication is generally an important practice for ourstudy population; however, whether and how discussions occur
are functions of multiple individual and dyadic level factors.
Inability to validate disclosed STI testing information, low
personal agency for initiating discussions, nonawareness of STI
testing talks as a health practice, and low discussion valuation
related to partner-type or condom use may inhibit STI healthcommunication from occurring. Electronic PHR access for STI
screenings pose a viable solution to barriers preventing STI
talks. Participants anticipate that access will be accompanied
by testing discussions earlier in relationships and more
frequently occurring prior to sexual encounters. Electronic PHRs
are expected to add novel validation to screening information
shared by partners and make it easier to initiate conversations.
Male participants were more likely to believe electronic PHRs
improve self-efficacy for discussions and in their ability to sharepositive results. Thus, electronic PHRs additionally offer new
avenues for increasing male participation in STI prevention.
Self-reported behaviors potentiating STI transmission such as:
sex without discussing STI testing, partner concurrency, sex
while intoxicated, and condom-less sex practices is evidenceof the need to continue targeting young black populations for
STI interventions. Our study supports an increasing amount ofburgeoning research on the feasibility and acceptability ofdelivering effective sexual health interventions through weband mobile-based platforms [17,19,30-32]. Furthermore, it
provides new insight into the role of patient electronic accessin improving dyadic communication on STI risk. Participants
almost ubiquitously anticipate that refusing to share STI
screening results will warrant partner suspicion. These beliefs
perhaps indicate that electronic PHR access will influence
decisions related to sexual behavior.
http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

Limitations
This study has many strengths and limitations. Our findings
offer rich data on electronic PHR access beliefs within our
sample population in context of when, how, and why STI talks
occur. The mixed-methods design allowed us to formatively
identify important variables to study quantitatively for a novel
practice. Similarly, the context of perceptions emerging fromour study is largely in absence of prior participant exposure toelectronic PHR access. Resources were not available for research
assistants nor secondary coders; thus, qualitative findings lackinter-coder reliability and are therefore subject to researcher
biases. Additionally, significant differences were observed inacademic classification by gender in our non-random
convenience sample of online survey participants.
Although we determined that many participants reported
electronic PHR access would lead to earlier dyadic talks on STI
prevention, future studies are needed to better understandwhether electronic PHR access would truly extend proximity
in time between STI talks and sexual encounters. Furthermore,
to determine whether an increase in time between the two events
minimizes the length of time to next STI screening between
dyad members.
Low self-efficacy beliefs for sharing positive electronic results
are likely an indicator of stigma associated with being diagnosed
with an STI. Given the sample, our study does not provide
substantial insight into perceptions about electronic
PHR–facilitated STI talks among people with chronic infections
like HIV and genital herpes. Nevertheless, reducing stigma andenabling individuals infected with STIs to safely andcomfortably disclose infection status to partners remains animportant challenge to prevention and care. Reducing stigma
associated with discussing infection with partners may reducebehaviors that accompany non-disclosure of diagnosis, such as
condom-less sex; in addition to stigma-related impacts on the
HIV treatment cascade [33-36]. Future studies are needed to
explore feasibility of addressing stigma-related outcomes withelectronic PHRs.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.26
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Conclusions
mHealth interventions incorporating electronic PHRs will offer
new insight into strengthening infrastructure and the capacity
to target disparities in STIs. Findings suggest that access toelectronic PHRs for STI screening among subpopulations ofblack youth may improve both motivation and personal agency
for initiating dyadic talks about testing. Results from this study
will likely inform novel interventions that use access toelectronic PHRs to stimulate important health-related
discussions between sexual partners. The preventative capacityof electronic PHRs envisioned by our sample cannot be achieved

Jackman et al
without policies that support equipping them with patient portal
access to STI screening records. Messages presented byhealthcare providers on adopting electronic PHR–delivered STI
results and electronic PHR–facilitated risk discussions willundoubtedly be key in adoption decisions. Moving forward
requires studying strategies for implementing interventions thatleverage electronic PHRs to create new sexual healthcommunication channels with providers, peers, and family
among black youth. With anticipated proliferation of electronic
PHR adoption in generations to come, close attention is needed
to ensure that black youth have equitable healthcare access toquality electronic PHR services [36].

Acknowledgments
The eSHINE Study was supported by a 2014-2016 dissertation research grant (R36HS023057) from the Agency for Healthcare
Research and Quality (AHRQ). R36HS023057 Project Mentor, Farin Kamanagar, MD, PhD, MPH (Associate Dean for Research,
Morgan State University (MSU), School of Computer, Mathematical and Natural Sciences) and Dissertation Committee Chair,
Bazle Mian Hossain PhD, MHS, MS (Professor, MSU, School of Community Health and Policy). Manuscript development was
funded by a 2017-2019 T32 NRSA Postdoctoral Training Fellowship in HIV Epidemiology and Prevention Sciences(2T32AI102623-06) within the Johns Hopkins University Center for Public Health and Human Rights. SDB’s effort was fundedin part from the Johns Hopkins University Center for AIDS Research, an NIH funded program (P30AI094189), which is supportedby the following NIH Co-Funding and Participating Institutes and Centers: NIAID, NCI, NICHD, NHLBI, NIDA, NIMH, NIA,
FIC, NIGMS, NIDDK, and OAR. The content is solely the responsibility of the authors and does not necessarily represent theofficial views of the NIH nor AHRQ.

Conflicts of Interest
None declared.

References
1.

2.

3.
4.

5.
6.

7.

8.

9.

10.

Centers FDC. Sexually Transmitted Disease Surveillance. Atlanta: US. Department of Health and Human Services; 2017.
URL: https://www.cdc.gov/std/stats16/CDC_2016_STDS_Report-for508WebSep21_2017_1644.pdf [accessed 2018-05-14][WebCite Cache ID 6zPgO7eKP]
Centers for Disease Control and Prevention. HIV Surveill Rep. Atlanta, GA; 2017. Diagnoses of HIV Infection in the
United States and Dependent Areas, 2016 URL: https://www.cdc.gov/hiv/library/reports/hiv-surveillance.html [accessed
2018-05-15] [WebCite Cache ID 6zR2CtT8P]
US Census Bureau. WebCite Original. The Black Alone Population in the United States URL: https://census.gov/data/tables/2014/demo/race/ppl-ba14.html [accessed 2018-05-15] [WebCite Cache ID 6zR36xVY6]Friedman AL, Bloodgood B. “Something We'd Rather Not Talk About”: findings from CDC exploratory research on
sexually transmitted disease communication with girls and women. J Womens Health (Larchmt) 2010 Oct;19(10):1823-1831.
[doi: 10.1089/jwh.2010.1961] [Medline: 20929416]
Centers for Disease ControlPrevention. Start Talking Stop HIV. URL: https://www.cdc.gov/actagainstaids/campaigns/
starttalking/ [accessed 2018-05-15] [WebCite Cache ID 6zR3LCc2V]
McFarlane M, Brookmeyer K, Friedman A, Habel M, Kachur R, Hogben M. GYT: Get Yourself Tested Campaign Awareness:
Associations With Sexually Transmitted Disease/HIV Testing and Communication Behaviors Among Youth. Sex Transm
Dis 2015 Nov;42(11):619-624. [doi: 10.1097/OLQ.0000000000000361] [Medline: 26457487]
Wilkerson JM, Fuchs EL, Brady SS, Jones-Webb R, Rosser BRS. Correlates of human immunodeficiency virus/sexuallytransmitted infection (HIV/STI) testing and disclosure among HIV-negative collegiate men who have sex with men. J Am
Coll Health 2014;62(7):450-460 [FREE Full text] [doi: 10.1080/07448481.2014.917654] [Medline: 24794417]
Thomas PE, Voetsch AC, Song B, Calloway D, Goode C, Mundey L, et al. HIV risk behaviors and testing history inhistorically black college and university settings. Public Health Rep 2008;123 Suppl 3:115-125 [FREE Full text] [doi:10.1177/00333549081230S314] [Medline: 19166095]
Hou S. HIV-related behaviors among black students attending Historically Black Colleges and Universities (HBCUs) versus
white students attending a traditionally white institution (TWI). AIDS Care 2009 Aug;21(8):1050-1057. [doi:10.1080/09540120802626196] [Medline: 20024762]
Green J, Ferrier S, Kocsis A, Shadrick J, Ukoumunne OC, Murphy S, et al. Determinants of disclosure of genital herpes
to partners. Sex Transm Infect 2003 Feb;79(1):42-44 [FREE Full text] [Medline: 12576613]

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.27
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
11.

12.

13.
14.
15.

16.

17.

18.

19.

20.

21.
22.
23.
24.
25.

26.
27.
28.
29.
30.

31.

32.

33.

34.

Xia Q, Molitor F, Osmond DH, Tholandi M, Pollack LM, Ruiz JD, et al. Knowledge of sexual partner's HIV serostatus
and serosorting practices in a California population-based sample of men who have sex with men. AIDS 2006 Oct
24;20(16):2081-2089. [doi: 10.1097/01.aids.0000247566.57762.b2] [Medline: 17053354]
Lunze K, Cheng D, Quinn E, Krupitsky E, Raj A, Walley A, et al. Nondisclosure of HIV infection to sex partners andalcohol's role: a Russian experience. AIDS Behav 2013 Jan;17(1):390-398. [doi: 10.1007/s10461-012-0216-z] [Medline:
22677972]
White H. -02-01. 2018. National HIV/AIDS Strategy for the United States: Updated to 2020 URL: https://files.hiv.gov/
s3fs-public/nhas-update.pdf [accessed 2018-02-01] [WebCite Cache ID 6wu21qtot]
LeGrand S, Muessig KE, Horvath KJ, Rosengren AL, Hightow-Weidman LB. Using technology to support HIV self-testing
among MSM. Curr Opin HIV AIDS 2017 Sep;12(5):425-431. [doi: 10.1097/COH.0000000000000400] [Medline: 28617712]
Muessig KE, LeGrand S, Horvath KJ, Bauermeister JA, Hightow-Weidman LB. Recent mobile health interventions to
support medication adherence among HIV-positive MSM. Curr Opin HIV AIDS 2017 Sep;12(5):432-441. [doi:10.1097/COH.0000000000000401] [Medline: 28639990]
Horvath KJ, Lammert S, LeGrand S, Muessig KE, Bauermeister JA. Using technology to assess and intervene with illicit
drug-using persons at risk for HIV. Curr Opin HIV AIDS 2017 Sep;12(5):458-466. [doi: 10.1097/COH.0000000000000398][Medline: 28771449]
Holloway I, Winder T, Lea CI, Tan D, Boyd D, Novak D. Technology Use and Preferences for Mobile Phone-Based HIV
Prevention and Treatment Among Black Young Men Who Have Sex With Men: Exploratory Research. JMIR Mhealth
Uhealth 2017 Dec 13;5(4):e46 [FREE Full text] [doi: 10.2196/mhealth.6436] [Medline: 28408360]
Yang C, Linas B, Kirk G, Bollinger R, Chang L, Chander G, et al. Feasibility and Acceptability of Smartphone-Based
Ecological Momentary Assessment of Alcohol Use Among African American Men Who Have Sex With Men in Baltimore.
JMIR Mhealth Uhealth 2015 Jun 17;3(2):e67 [FREE Full text] [doi: 10.2196/mhealth.4344] [Medline: 26085078]
Widman L, Golin CE, Noar SM, Massey J, Prinstein MJ. ProjectHeartforGirls.com: Development of a Web-Based HIV/STD
Prevention Program for Adolescent Girls Emphasizing Sexual Communication Skills. AIDS Educ Prev 2016
Oct;28(5):365-377 [FREE Full text] [doi: 10.1521/aeap.2016.28.5.365] [Medline: 27710087]
US Congress. Health Information Technology for EconomicClinical Health Act. 2009. Title XIII of Division A and Title
IV of Division B of the American Recovery and Reinvestment Act of 2009 URL: https://www.hhs.gov/sites/default/files/
ocr/privacy/hipaa/understanding/coveredentities/hitechact.pdf [accessed 2018-05-15] [WebCite Cache ID 6zR4mNuS6]
Klassen AC, Creswell J, Plano CVL, Smith KC, Meissner HI. Best practices in mixed methods for quality of life research.
Qual Life Res 2012 Apr;21(3):377-380. [doi: 10.1007/s11136-012-0122-x] [Medline: 22311251]
Creswell J, Plano CV. Designing and Conducting Mixed Methods Research. 2nd Edition. Los Angeles, CA: SAGE
Publications; 2011.
Rogers E. Diffusion of Innovations, 5th Edition. New York, New York: Free Press; 2003.
Fishbein M, Yzer M. Using Theory to Design Effective Health Behavior Interventions. Communication Theory
2003;13(2):164-183.
Chaudoir SR, Fisher JD, Simoni JM. Understanding HIV disclosure: a review and application of the Disclosure Processes
Model. Soc Sci Med 2011 May;72(10):1618-1629 [FREE Full text] [doi: 10.1016/j.socscimed.2011.03.028] [Medline:
21514708]
ATLAS.ti [computer software]. Version 7.1.8. Berlin, Germany: Scientific Software Development; 2014.
Qualtrics [computer software]. Version January-May 2015. Provo, Utah: Qualtrics; 2015.
Glanz K, Rimer B, Viswanath K. Health Behavior and Health Education. Fourth Edition. (Glanz K, Rimer B, ViswanathK. eds.). San Francisco, CA: Jossey-Bass; 2008.
STATA [computer software]. Version 14. College Station, TX: StataCorp; 2015.
Veinot TC, Campbell TR, Kruger DJ, Grodzinski A. A question of trust: user-centered design requirements for an informatics
intervention to promote the sexual health of African-American youth. J Am Med Inform Assoc 2013;20(4):758-765 [FREEFull text] [doi: 10.1136/amiajnl-2012-001361] [Medline: 23512830]
Canidate S, Hart M. The Use of Avatar Counseling for HIV/AIDS Health Education: The Examination of Self-Identity in
Avatar Preferences. J Med Internet Res 2017 Dec 01;19(12):e365 [FREE Full text] [doi: 10.2196/jmir.6740] [Medline:
29196281]
Cao B, Gupta S, Wang J, Hightow-Weidman LB, Muessig KE, Tang W, et al. Social Media Interventions to Promote HIV
Testing, Linkage, Adherence, and Retention: Systematic Review and Meta-Analysis. J Med Internet Res 2017 Nov
24;19(11):e394 [FREE Full text] [doi: 10.2196/jmir.7997] [Medline: 29175811]
Camacho-Gonzalez AF, Wallins A, Toledo L, Murray A, Gaul Z, Sutton MY, et al. Risk Factors for HIV Transmission
and Barriers to HIV Disclosure: Metropolitan Atlanta Youth Perspectives. AIDS Patient Care STDS 2016 Jan;30(1):18-24[FREE Full text] [doi: 10.1089/apc.2015.0163] [Medline: 26588663]
Santos-Hövener C, Zimmermann R, Kücherer C, Bätzing-Feigenbaum J, Wildner S, Hamouda O, et al. Conversation about
Serostatus decreases risk of acquiring HIV: results from a case control study comparing MSM with recent HIV infectionand HIV negative controls. BMC Public Health 2014 May 14;14:453 [FREE Full text] [doi: 10.1186/1471-2458-14-453][Medline: 24885694]

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

Jackman et al

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.28
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
35.

36.

Jackman et al

Gallo MF, Margolis AD, Malotte CK, Rietmeijer CA, Klausner JD, O'Donnell L, Safe in the City Study Group. Sexual
abstinence and other behaviours immediately following a new STI diagnosis among STI clinic patients: Findings from the
Safe in the City trial. Sex Transm Infect 2016 May;92(3):206-210. [doi: 10.1136/sextrans-2014-051982] [Medline: 26670913]
Ford EW, Hesse BW, Huerta TR. Personal Health Record Use in the United States: Forecasting Future Adoption Levels.
J Med Internet Res 2016 Mar 30;18(3):e73 [FREE Full text] [doi: 10.2196/jmir.4973] [Medline: 27030105]

Abbreviations
CDC: Centers for Disease Control & Prevention
CMS: Centers for Medicare & Medicaid Services
DOI: Diffusion of Innovation Theory
eSHINE: Electronic Sexual Health Information Notification and Education
HITECH: Health Information Technology for Economic and Clinical Health Act
HSV-2: herpes simplex virus-2
IMBP: Integrative Model of Behavioral Prediction
MSM: men who have sex with men
PHR: personal health record
PI: principal investigator
STI: sexually transmitted infection

Edited by G Eysenbach; submitted 21.10.17; peer-reviewed by HJ Rendina, R Robbins, M Gallo, T Winder; comments to author
07.12.17; revised version received 01.02.18; accepted 15.04.18; published 12.07.18.
Please cite as:Jackman KM, Baral SD, Hightow-Weidman L, Poteat T
Uncovering a Role for Electronic Personal Health Records in Reducing Disparities in Sexually Transmitted Infection Rates Among
Students at a Predominantly African American University: Mixed-Methods StudyJMIR Med Inform 2018;6(3):e41
URL: http://medinform.jmir.org/2018/3/e41/
doi:10.2196/medinform.9174
PMID:30001998©Kevon-Mark Jackman, Stefan David Baral, Lisa Hightow-Weidman, Tonia Poteat. Originally published in JMIR Medical
Informatics (http://medinform.jmir.org), 12.07.2018. This is an open-access article distributed under the terms of the Creative
Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The
complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright andlicense information must be included.

http://medinform.jmir.org/2018/3/e41/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e41 | p.29
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Risling et al

Original Paper

Defining Empowerment and Supporting Engagement Using Patient
Views From the Citizen Health Information Portal: Qualitative
Study
Tracie Risling1*, RN, PhD; Juan Martinez1*, RN, BSN, BSc; Jeremy Young1*, RN, BSN, BSc; Nancy Thorp-Froslie1*,
BS, MPH
College of Nursing, University of Saskatchewan, Saskatoon, SK, Canada*

all authors contributed equally

Corresponding Author:
Tracie Risling, RN, PhD
College of Nursing
University of Saskatchewan
4218 Health Sciences Building, E-Wing
104 Clinic Place
Saskatoon, SK, S7N 2Z4
CanadaPhone: 1 306 966 6232
Email: tracie.risling@usask.ca

Abstract
Background: The increasing presence of technology in health care has created new opportunities for patient engagement andwith this, an intensified exploration of patient empowerment within the digital health context. While the use of technology, suchas patient portals, has been positively received, a clear linkage between digital health solutions, patient empowerment, and healthoutcomes remains elusive.
Objective: The primary objective of this research was to explore the views of participants enrolled in an electronic health record
portal access trial regarding the resultant influence of this technology on their feelings of patient empowerment.
Methods: The exploration of patient empowerment within a digital health context was done with participants in a tetheredpatient portal trial using interpretive description. Interpretive description is a qualitative methodology developed to pragmaticallyaddress clinical health questions. Patient demographics, self-reported health status, and self-identified technology adaptation
contributed to the assessment of empowerment in this qualitative approach.
Results: This research produced a view of patient empowerment within the digital health context summarized in two overarchingcategories: (1) Being Heard and (2) Moving Forward. In each of these, two subcategories further delineate the aspects of
empowerment, as viewed by these participants: Knowing More and Seeing What They See under Being Heard, and Owning
Future Steps and Promoting Future Care under Moving Forward. This work also highlighted an ongoing interconnectedness
between the concepts of patient empowerment, engagement, and activation and the need to further articulate the unique aspectsof each of these.
Conclusions: The results of this study contribute needed patient voice to the ongoing evolution of the concept of patient
empowerment. In order to move toward more concrete and accurate measure of patient empowerment and engagement in digital
health, there must be further consideration of what patients themselves identify as essential aspects of these complex concepts.
This research has revealed relational and informational elements as two key areas of focus in the ongoing evolution of patient
empowerment operationalization and measure.
(JMIR Med Inform 2018;6(3):e43) doi:10.2196/medinform.8828
KEYWORDS
digital health; electronic health record; patient engagement; patient empowerment; patient portal

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.30
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Introduction
Patient-centered care has long been promoted, debated, andpursued, yet substantial reform to established institutional
approaches has proved challenging. The increasing presence oftechnology in health care delivery, coupled with consumer
demand, is contributing to a remarkable shift in traditional
practices and a renewed interest in patient empowerment as ameans to advance this long-sought reform [1,2]. However,decades of debate and varied conceptual application haveresulted in a lack of clarity in how to best operationalize or evenconsistently define patient empowerment [3-6]. Despite thesechallenges, recent reviews on patient empowerment reveal a
global interest in this concept [2,3,5]. The World HealthOrganization (WHO) European Regional Office included
empowerment and patient-centered practice as key elements in
its Health 2020 report [7], a follow-up on previous WHO study
on the effectiveness of empowerment to improve health [8]. In
this earlier work, WHO identified empowerment as an essential
public health strategy and also noted a scarcity of refinedevaluative measures [8]. Many years later, this need for acomprehensive operational definition of patient empowerment
and robust measures to evaluate the concept remains [2,3,5].
Publication on patient empowerment is increasingly present indigital health literature [9-12], with an emerging considerationthat “the future of patient empowerment may lie in technologicaladvancements and better access of patients to thesetechnologies” [13]. As technology is promoted as a means toadvance patient empowerment [9,10,14,15], the need to addresscontextual considerations in applying this concept to digital
health has also been raised [12,16]. Patient empowerment has
been examined in conjunction with technological apps such as
electronic personal health records [17,18], patient Web portals[19], and electronic medical records [20]. Patient portals or“tethered” electronic personal health records are Web-based
portals linked to electronic health records (EHRs) [14]. Studies
on patient portals seem to have emerged as a focal point in thisstudy, with connections made between portal use, patient
empowerment, engagement, and activation and ultimately
improved personal health outcomes [21-24].
Despite this promising beginning in the exploration of patient
empowerment in digital health, a clear linkage among the useof digital health services, patient empowerment, and healthoutcomes remains elusive. Previous work has characterized thechallenges of advancing a clear and comprehensive definitionof patient empowerment in digital health app [3-5]. Suchchallenges emerge from the mixed or combined use of patient
empowerment with terms such as patient engagement,
enablement, activation, and even patient-centeredness, although
calls for the distinct use and application of each of theseconceptual entities have repeatedly been made [5,25,26]. A
previously published scoping review, by these authors on patient
empowerment measures, further highlighted the current discordin operationalizing patient empowerment [27]. The gaps andinconsistencies in the measures of empowerment are further
exacerbated by negligible contributions of patient voice [3,27].
As such, incorporating patient perceptions into the developmentof patient empowerment measures has emerged as a critical
http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

Risling et al
need to disentangle the interconnectedness between patient
empowerment, engagement, and activation, specifically withregards to the use of patient portals in digital health [3,27]. The
primary objective of this qualitative study was to explore theinfluence of the patient portal use on patient views andperceptions of empowerment. Incorporating an underrepresented
qualitative narrative with the current empowerment study
maximizes opportunities for patient voice to direct
patient-centered care while contributing to the needed
delineation of empowerment aspects associated with patient
portal use.

Methods
Study Aim
This study aimed to characterize participant experiences andviews of empowerment related to the use of patient portals. This
study was supported by a previous scoping review, by theseauthors, which explored current practices in theoperationalization of patient empowerment in relation to theuse of tethered patient portals [27].

Recruitment, Appraisal of Health, and Technology
Adaptation
Following the approval from the Research Ethics Board of thelead author’s institution, participant recruitment was conductedin collaboration with eHealth Saskatchewan, which, with thesupport of Canada Health Infoway, had recently deployed alimited launch of the Citizen Health Information Portal (CHIP).
This tethered patient portal was provided to approximately 1000
provincial residents facilitating access to data contained in their
EHR. The portal allowed patients to add medical history
information and provided views of laboratory results,
immunizations, prescriptions, and hospital discharge summaries,
as well as an opportunity to set reminders for medications andappointments [28]. All CHIP participants received an email
invitation to participate in the study. No geographical limits
were imposed during recruitment, but participants did need to
be English speakers. The recruitment process resulted in apurposive sample of 26 participants for the study.
Regarding the demographic data collection, participants wereasked to self-identify their current health status. The primarymeans of accessing Web-based information was also assessedby self-report in this group. Finally, the research team used amodified adaptation of Rogers’ diffusion of innovations theory[29] to explore participant comfort in adopting and using newtechnologies. Rogers used the following 5 stages to identify
how innovations are transmitted and taken up by members ofa social system: (1) Innovators, these individuals are typicallyrisk-takers and lead the way when it comes to new technologies;(2) Early Adopters, includes individuals who will be among thefirst to try new technologies; (3) Early Majority, represents
individuals who take time to consider trying out something new
before acquiring and using it; (4) Late Majority, includes thegroup of individuals that is somewhat cautious about trying newtechnologies and will tend to adopt the use of technology more
slowly than the average; and (5) Questioners, representing thelast group that will uptake a new technology with users that
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.31
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
typically require proof that new technology is worth investingin [29].

Data Collection and Analysis
Data collection was primarily achieved through semistructured
interviews (Multimedia Appendix 1), and members of theresearch team also observed and collected data during a CHIP
pilot participant focus group run by eHealth Saskatchewan.
Once complete, the digital data files were transcribed,
deidentified, and reviewed for accuracy, in preparation foranalysis.
In this study, interpretive description (ID) was used, a qualitative
methodology first detailed by Thorne et al [30]. ID wasdeveloped as a way of generating clinically relevant knowledge
for health disciplines, and Thorne has since published, andrecently updated, full text on this approach [31]. Initially, adetailed line-by-line coding of the data was undertaken within
the transcript documents to provide an initial sense of the scopeof the data. These early coding efforts revealed a wide diversity
in the topic, enhanced by the line-by-line approach. The researchteam returned to the transcriptions to create a more focused
dataset by collating all passages related to empowerment andengagement. In addition, the study team included any mentionof either, or both, of these terms in the dataset based on theestablished interconnection of the concepts in the previously
conducted scoping review [27].
The dataset was prepared in a split-page word document format,
and the research team returned to the coding process. Moving
away from the granular line-by-line approach, the data wereinstead examined, as Thorne recommended, through an“exploration of commonalities and differences among andbetween individual experiences” [32], reflective memoing wasused during the analysis along with regular interchanges between
research team members to share emerging interpretations andcode categories. The ongoing dialogue supported the explorationof outlier data and allowed the team to come to consensus on

Risling et al
how to integrate these findings. As the code categories began
to coalesce and the interpretation emerged, the research teamused the proposed descriptive categories and returned to the fulltranscript data set for a final review; this was done to ensure noother experiences or elements needed to be integrated into theanalysis and final interpretation.

Results
Participant Demographics
All participants (N=26) resided in one Western Canadianprovince. The majority of participants were females (n=18),
with fewer (n=8) male participants. Although the age ofparticipants ranged 20-85 years, the predominant age category
was 60-69 years (n=14), as shown in Figure 1.
As summarized in Figure 2, there was a diversity of self-reported
health status, with the majority of participants rating their healthwithin the well options. The way through which participants
gained access to their health information on the portal varied,with most using a combination of personal computers, tablets,
and mobile devices (n=15; data not shown). Figure 3 shows therange of self-identified technology adoption among the study
participants, spanning the full scope of Rogers’ continuum.

Interpretive Description Findings
Categories
From the experiences shared by CHIP participants (N=26) inthis qualitative exploration of empowerment and engagement,
the following two overarching categories emerged: Being Heard
and Moving Forward. For each of these, 2 subcategories were
defined to delineate the aspects of empowerment andengagement further, as viewed by these participants: Knowing
More and Seeing What They See (under Being Heard) andOwning Future Steps and Promoting Future Care (under Moving
Forward). Figure 4 presents these categories, and each will be
reviewed further.

Figure 1. Participants’ age ranges.

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.32
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Risling et al

Figure 2. Self-identified health status of participants ranging from well to acutely ill with chronic illness(es).

Figure 3. Technology adoption by participants.

ID is especially useful in considering the voice of the outlier orcontrasting case [31]. In this way, ID is an opportunity tooperationalize a patient-centered approach in the research app.
In the case of this interpretive work on empowerment andengagement in relation to the patient portal experience, theresearch team encountered a participant whose view of
empowerment was contrary to what had been commonly
expressed. These data supported the emergence of the firstoverarching category in this interpretation, Being Heard.

Being Heard
Very early in the exploration of the participant transcripts, strong
support emerged for empowerment, and many positive views
about the concept were expressed. However, 1 participant did
http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

not seem to share this certainty when asked about the conceptof patient empowerment, he responded:
I don’t know if I like that word. It is just encouraging
to see that healthcare is going in a positive direction
these days, and it just tells me and it makes me feellike I do have a say in my healthcare and how it’sadministered to me and I want people to hear me thefirst time I present them with an issue, and that’s
something I really appreciate.
Although the participant was unsure about the use of patient
empowerment, there existed an essential element that arosefrom his statement about being heard the first time that resonatedwith the research team. A code category Being Heard was
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.33
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Risling et al

proposed, and the team returned to the data using the lens of
this outlier view to determine whether other participants
expressed similar sentiments. From this exploration, theoverarching category of Being Heard was established. During
the interpretative analysis, this category evolved to include 2
further subcategories or themes: Knowing More and Seeing
What They See. Table 1 summarizes participant quotes
supporting the development of this category and the included
themes.

Together, Knowing More and Seeing What They See represent
the empowered view of Being Heard. Overwhelmingly, study
participants shared views that demonstrated that they werelooking for opportunities to not only be further engaged in theirown health care but also feel as though they were valued andknowledgeable team members in the decision making anddelivery of this care. This sense of empowerment in future
direction was represented in the second category of Moving
Forward.

Knowing More
Pilot participants in the CHIP project could access their EHR,
and a familiar sentiment about the availability of this new sourceof personal data was expressed by 1 study participant: “I guess
I could go back to the old adage where you hear people say
information is power.” In general, there was a significant
response to the value of having direct access to personal healthcare data, especially as laboratory and medication information
was updated in near real-time. This knowledge was often
connected to the sense of empowerment, as demonstrated in thequoted excerpts in Table 1. Furthermore, the access to
information was a foundational element of empowerment forthese participants and a primary driver for their ongoinginteractions with the portal. Moreover, Knowing More was a
crucial foundation to support the pursuit of Being Heard.

Moving Forward

Seeing What They See
The second element of Being Heard was a focus on what could
be seen. Moving beyond the experience of simply Knowing
More, Seeing What They See reflected not only the information
but also an empowered sense of shared access to timely healthcare data with providers themselves. This access supported avariety of patient initiatives, as detailed in a few patient excerpts
highlighting this theme in Table 1. Although some mixed
feelings were reported regarding having access to test results,
as well as differing opinion on whether this would increase or
decrease physician workload, the overall sense of equality inbeing provided the same type of information that their own
providers received was remarkable for study participants. There
were recommendations that additional information, or
suggestions, from providers to support self-care or behavioral
changes, based on the data available in the portal, beincorporated in future offerings.

This second overarching categorization comprised 2
subcategories or themes, Owning Future Steps and Promoting
Future Care; these summarize the views of participants first in
relation to their own self-care and second regarding their hopesand expectations for future technological advancements to
support their ongoing involvement and engagement in theirhealth care. Table 1 details participant quotes supporting thiscategory and resultant themes.
Owning Future Steps
Although many participants in this study reported a history of
substantial involvement in directing their own health, there wasa profound sense of the significance of this portal technology
in cementing the role of patients in the care relationship,especially in relation to decision making and engaging in
self-care behaviors, as depicted in several quotes in Table 1.
Where Being Heard was about taking in information and feelingempowered in the patient-provider relationship, Moving
Forward, and in particular, Owning Future Steps, represented
the resulting action when patients worked from an empowered
position. Finally, the significance of the portal access in the
lives of these participants, and its influence on their views ofan evolving digital health care landscape, was summed in
Promoting Future Care.
Promoting Future Care
Participants in this study provided a powerful contrasting viewof a commonly held perception about the willingness of older
patients to engage in technological health care solutions.
Promoting Future Care not only highlights the desire of theseparticipants to have continued access to the citizen portal but
also for other technologies that could provide a more connected
and supported health care future; details of this can be seen in
the participant quotes for this theme in Table 1.

Figure 4. Interpretive categories identified in the qualitative study.

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.34
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Risling et al

Table 1. Participant quotations supporting the themes in the category Being Heard.
Category and theme

Quotations

Being Heard
Knowing More

•
•
•
•
•
•
•
•

Seeing What They See

•
•
•
•

“I mean when I say empower, it empowers me, I would say it just gives me the confidence to know whether I’masking the right questions or I have asked the right questions.”“It (the CHIP information) makes me a little bit more empowered to help make those decisions.”“Empowerment, so you can understand and know what’s going on.”“To be empowered means to be aware of what is happening with your health.”“I can look up and see that information, make decisions, have less repetitive questions with my doctor and focus
on the things that I really need to know.”“It’s helping me assume responsibility and to be knowledgeable and I think to just be better prepared when I go
see the doctor so I can ask meaningful questions.”“You’re not anxious about it and, you know, if you’re worried about something, you have a resource now thatkind of tells you where to go with it or helps you determine where to go with it.”“I think it provides me with some peace of mind, because like I said, instead of hoping and being pretty sure it’s
all fine, you can look, and you can know.”“If I can see what they are seeing maybe in a little bit in advance, then I can be better prepared when I go to my
doctor’s office; like have my list of questions ready.”“I go back and look at my numbers, try to make some changes right there. So it gives me more up-to-date informationand not having to wait to get in to see the doctor if there was something that’s off. If it’s something that I’d try tocontrol myself like the blood sugar levels or A1C then that’s something that I can take into my own consideration.”“CHIP has literally been a lifesaver for me, because I’ve been able to get my results before I even see the doctors,
I’m able to formulate the questions that I need to ask the doctor and get far more engaged in my health care.”“To be empowered means to be aware of what is happening with your health, and test results can help that. Information from doctors and health care people can help that, and that way you can make the changes and adjustmentsthat you may need to be well.”Moving Forward
Owning Future Steps

•
•
•
•
•

Promoting Future Care •
•
•
•
•
•
•
•
•

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX“When I become engaged in my healthcare and utilizing the resources I can access, I try to garner enough knowledge
and understanding to feel empowered to take the next step.”“It has become very apparent the importance of informed self-care, not taking the doctors and the health professionals out of the equation, but to be able to take some of that information and look at what I can do as an individual to help myself.”“I need to be able to manage that, because I’m the one that’s doing it every day.”“I can see that it’s vital, the way they we’re moving, that we have that self-empowerment that comes from using
this information, having access and using this information ourselves.”“Well, I think something like that will give you more empowerment because it’ll give you a track to follow, it
should give you something to follow, and I think that there’s even something else that could be done with that, in
terms of empowering your healthcare.”“I think if you want people engaged in the health system, you need to make the health system accessible, and I
think this is certainly one way to do it.”“I think this type of technology will also empower physicians to be able to give patients better care, or more
timely care.”“So I think that when you talk about technology and what it means, I think it’s just endless. There’s so much andI think it really has a future that will allow for a whole different kind of coordination and communication.”“CHIP is wonderful. I want very much for it to continue and I would encourage over time for more information
to be shared on it, I mean things like perhaps radiologist reports and those kinds of things.”“I think it’s a great idea, I really do. We live in a technology age, we’re in an online age. People want information
at their fingertips, they do a lot of this kind of stuff from home or from their smartphone or whatever. I think thisis an idea whose time is here.”“I personally think it’s amazing, obviously, it has its limitations, but the only challenge will be presenting it in a
way that people with different demographics and all types of background will accept it and be on board.”“I have mixed feelings about the increasing presence [of technology]. I love the access, I like the idea that if I
needed care in a hurry people would have access to my information. I think that the more that happens the better,but the trade-off of that is the risk of somebody going in and looking at your information to use it to do you harm.”“Well, I’m probably as ambivalent as a lot of people are. Worrying a bit about security of information, enjoying
having the access to it but still kind of okay, is it secure? How secure is it?”“Technology is, I mean I see it as a double-edged sword. It’s lifesaving and also costly as heck. I’m sure this whole
CHIP thing doesn’t come for free but as a service I think it’s very—however much it costs— I’m hoping it’s not
too much that they don’t find that it’s not worthwhile, because I think that it is (worthwhile) if it’s affordable.”

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.35
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Risling et al

Textbox 1. Participant quotations identifying challenges or limitations with patient portal use.
•

“The amount of information that’s there, you know it’s going to be overwhelming for some people.”•“I don’t really think it changed anything, because I knew enough to ask.”•“I sometimes don’t think it’s always necessary for a doctor to give you that information unless there’s an interpretation required.”•“I don’t know that I want to be more engaged. I don’t know enough about it so I just sort of trust that my doctor’s doing what’s best.”•“The patient engagement is for people who want it, not everybody is going to want it.”Though enthusiastic about the integration of technology into
their health care futures, the study participants also expressedpractical views regarding the potential impact of this shift inaccess and delivery. Concerns about privacy, cost, and theimportance of continuing to focus on people in care wereexpressed. As has been noted, ID promoted the inclusion ofoutlier views in the interpretative process. Textbox 1 summarizes
some key contrasting participants’ views about the portal
technology and engagement in this type of access.
The use of ID serves as a very effective reminder for researchers,
practitioners, and digital solution providers that no single
initiative is going to meet every need regarding patientengagement and empowerment. The ongoing exploration ofviews, like those represented in Textbox 1, is key to recognizing
not only potential research limitations but also those in the larger
digital health milieu. Overall, a strong positivity associated withthe expanding role of technology in health care, especially forservices like patient portals, which allowed for improved patient
access to their health care data.

Discussion
Principal Findings
Although there has been a considerable amount of academic
exploration and publication on patient empowerment, a recent
analysis revealed that a clear definition of the concept is stilllacking [3,5,6]. In addition, there has been little consideration
about the influence digital health may have on key attributes ofpatient empowerment. Overall, these findings support an existing
challenge with the operationalization of this concept for use indigital health research. Following a previously published scoping
review on patient empowerment measure in digital health [27],this qualitative study was conducted to contribute the needed
patient voice to the ongoing discourse on empowerment indigital health and provide insight for the development of future
measures.
Elements of the aspects of empowerment identified in this study
through Being Heard and Moving Forward, and their respective
subcategories, can be noted in previously articulated views of
empowerment summarized by Bravo et al [3] in their recent
analysis of patient empowerment: first, in this widely useddefinition by Funnell et al [33] from the early 1990s, “We havedefined patient empowerment as the discovery and developmentof one’s inherent capacity to be responsible for one’s own life.
People are empowered when they have sufficient knowledge
to make rational decisions, sufficient control and resources toimplement their decisions, and sufficient experience to evaluate
the effectiveness of their decisions” and second in the work of
http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

Lau [34] “Patient empowerment begins with information andeducation and includes seeking out information about one’s
own illness or condition, and actively participating in treatment
decisions.”
The alignment of this qualitative interpretation with previous
work on conceptualizing patient empowerment is a positive
finding; however, it is important to note further complexities
that have been proposed through the views of these participants,
particularly those that relate to the digital health context.
Findings highlighted in Promoting Future Care speak to a desireby patients for the continued ability to be able to access theirdata electronically, as well as their hopes for the interchange ofdata in the future to support new means of communicating with
providers and ultimately more efficient care. However, there
are also unresolved concerns represented in these participant
views, regarding the privacy and security of information,technology costs, and for some, uncertainty about whether or
not the data are actually wanted.
A direct connection among digital health, patient empowerment,
and particular health outcomes have been difficult to clearly
demonstrate, with conflicting reports on the success ofinterventions summarized in several previous systematic reviews[35-41]. This challenge is reflected in this study as well incontrasting participant views on particular aspects of this digital
health solution. However, despite concerns regarding costs or
potential security risks associated with digital data, participantsin this study still overwhelmingly identified portal access as anempowering force in the management of their health; this isclearly reflected in Knowing More. The impact of having digitalaccess to critical health data appeared to be transformative forparticipants and supported enhanced preparation for practitioner
visits, as a well as an improved sense of well-being or “peaceof mind.”
CHIP participants identified a positive association between theoffered digital health solution and their sense of patient
empowerment, which may prompt further reflection on how alack of patient participation in empowerment conceptualizationand measure, as identified by Bravo et al [3], may havecontributed to the inconclusiveness of some digital healthempowerment study [35-41]. An enhanced understanding of
empowerment incorporating patient views is clearly needed to
produce a more robust conceptualization. Only when thisfoundational work is complete should attempts to advance theevaluation and measurement of this concept move ahead. With
these considerations, future measures should more accurately
demonstrate the effectiveness of digital health tools insupporting patient empowerment. This study has provided apiece of this foundation by using patient voice to provide a set
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.36
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSof conceptual elements from which the work of creating afulsome operationalization of patient empowerment within
digital health can begin.
The results of this study can be used to reflect on the aspects ofpatient portals that may have the most “empowering effects.”
In particular, the combined participant views presented in
Knowing More, Seeing What They See, and Owning Future
Steps lend support to features such as the timely release ofdiagnostic results and the ability of patients to contribute data
into their EHR. Evidence suggests that interventions, such as
patient portals, can improve self-efficacy by giving patients
tools for enhanced self-management, which, in turn, may
contribute to a heightened sense of patient empowerment [9,18].
This study further highlights the empowering sense of equalitythat results when patients feel they have access to the same dataas their health care providers, when they are Seeing What They
See. Several patients noted a change in the dynamics of theirprovider relationships and identified the timely delivery of
information as a critical element for their ongoing engagementin their health management. Furthermore, the ability to securely
exchange information through a portal was acknowledged as apriority for more active patient collaboration and provides
support to ongoing advocacy efforts for increasingly open data
adoption in digital health solutions.

Limitations

Risling et al
is not generalizable or representative of the experiences of all
CHIP users or of other patients who are engaged in tetheredpatient portal use. This interpretation of patient empowerment
within a digital health context, focused on the portal use, may
however resonate with other patients, practitioners, and service
providers. Certainly, it has added much needed patient voice tothe ongoing academic pursuit of the comprehensive
conceptualization of patient empowerment.

Conclusions
Providing patients electronic access to timely personal healthinformation is a crucial step in supporting a new era ofcollaborative care. In this study, the use of a tethered patient
portal (CHIP) introduced participants to the benefits oftechnology in supporting positive health outcomes and theirown empowerment. CHIP participants shared their views ofpatient empowerment situated within the context of this digital
health experience. This qualitative study produced a morein-depth patient-centric view of patient empowerment, including
additional considerations relevant to the operationalization ofthis concept within a digital health context. This study supports
the development of a patient empowerment measure to more
effectively capture the influence of digital health initiatives on
this vital outcome. The development of this instrumentation is
the primary objective of ongoing research by members of thisstudy team.

This study is situated in the context in which it was conducted
and represents the views of pilot CHIP participants. As such, it

Acknowledgments
The research team would like to thank the 26 study participants who volunteered their time to contribute to this work and provide
new insights into patient empowerment in digital health. The study team also acknowledges the funding provided by eHealth
Saskatchewan for this research, as well as the role of Canada Health Infoway in supporting the pilot delivery of the CHIP. In
addition, we extend our thanks to the University of Saskatchewan for the provision of funds associated with the undergraduate
summer research assistant program.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Semistructured interview guide.
[PDF File (Adobe PDF File), 55KB - medinform_v6i3e43_app1.pdf ]References
1.

2.

3.
4.

Snowdon A, Alessi C, Schnarr K. It's all about me: the personalization of health systems. 2014 URL: http://worldhealthinnovationnetwork.com/images/publications/summaries/
AllAboutMe-ThePersonalizationofHealthcareSystems_ExecutiveSummary.pdf [WebCite Cache ID 6n3qZfWeH]
Cerezo PG, Juvé-Udina M, Delgado-Hito P. Concepts and measures of patient empowerment: a comprehensive review.
Rev Esc Enferm USP 2016;50(4):667-674 [FREE Full text] [doi: 10.1590/S0080-623420160000500018] [Medline:
27680054]
Bravo P, Edwards A, Barr PJ, Scholl I, Elwyn G, McAllister M, et al. Conceptualising patient empowerment: a mixed
methods study. BMC Health Serv Res 2015;15:252 [FREE Full text] [doi: 10.1186/s12913-015-0907-z] [Medline: 26126998]
Herbert RJ, Gagnon AJ, Rennick JE, O'Loughlin JL. A systematic review of questionnaires measuring health-related
empowerment. Res Theory Nurs Pract 2009;23(2):107-132. [Medline: 19558027]

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.37
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
5.
6.
7.

8.
9.
10.

11.
12.

13.
14.
15.
16.
17.

18.

19.

20.

21.
22.

23.
24.

25.

26.
27.
28.

Barr PJ, Scholl I, Bravo P, Faber MJ, Elwyn G, McAllister M. Assessment of patient empowerment--a systematic review
of measures. PLoS One 2015;10(5):e0126553 [FREE Full text] [doi: 10.1371/journal.pone.0126553] [Medline: 25970618]
Cyril S, Smith BJ, Renzaho AMN. Systematic review of empowerment measures in health promotion. Health Promot Int
2016 Dec;31(4):809-826. [doi: 10.1093/heapro/dav059] [Medline: 26137970]
World Health Organization Regional Office for Europe. 2012. Health 2020: a European policy framework supporting action
across government and society for health and well-being URL: http://www.euro.who.int/__data/assets/pdf_file/0009/169803/
RC62wd09-Eng.pdf [WebCite Cache ID 6riDMJ5HB]
Wallerstein N. What is the evidence on effectiveness of empowerment to improve health?. 2006. URL: http://www.
euro.who.int/__data/assets/pdf_file/0010/74656/E88086.pdf [WebCite Cache ID 6riCvQHGr]
Calvillo J, Román I, Roa LM. How technology is empowering patients? A literature review. Health Expect 2013 May 28.
[doi: 10.1111/hex.12089] [Medline: 23711169]
Lettieri E, Fumagalli LP, Radaelli G, Bertele' P, Vogt J, Hammerschmidt R, et al. Empowering patients through eHealth:a case report of a pan-European project. BMC Health Serv Res 2015 Aug 05;15:309 [FREE Full text] [doi:10.1186/s12913-015-0983-0] [Medline: 26242863]
Hill R, Betts LR, Gardner SE. Older adults’ experiences and perceptions of digital technology: (Dis)empowerment, wellbeing,
and inclusion. Computers in Human Behavior 2015 Jul;48:415-423 [FREE Full text] [doi: 10.1016/j.chb.2015.01.062]
Bos L, Marsh A, Carroll D, Gupta S, Rees M. Patient 2.0 Empowerment. 2008 Presented at: 2008 International Conference
on Semantic Web & Web Services; 2008; Las Vegas URL: http://www.p4f.be/P4F/_images/
20100408patientempowermenthealth20.pdf
Kambhampati S, Ashvetiya T, Stone NJ, Blumenthal RS, Martin SS. Shared Decision-Making and Patient Empowerment
in Preventive Cardiology. Curr Cardiol Rep 2016 May;18(5):49. [doi: 10.1007/s11886-016-0729-6] [Medline: 27098670]
Irizarry T, DeVito DA, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet
Res 2015;17(6):e148 [FREE Full text] [doi: 10.2196/jmir.4255] [Medline: 26104044]
Lober WB, Flowers JL. Consumer empowerment in health care amid the internet and social media. Semin Oncol Nurs
2011 Aug;27(3):169-182. [doi: 10.1016/j.soncn.2011.04.002] [Medline: 21783008]
Wickramasinghe N, Bali R, Suomi R, Kirn S, editors. Patient Empowerment: A Two Way Road. In: Critical Issues for theDevelopment of Sustainable E-Health Solutions. Boston: Springer; 2012.
Crouch PB, Rose CD, Johnson M, Janson SL. A pilot study to evaluate the magnitude of association of the use of electronic
personal health records with patient activation and empowerment in HIV-infected veterans. PeerJ 2015;3:e852 [FREE Fulltext] [doi: 10.7717/peerj.852] [Medline: 25802815]
Tuil WS, Verhaak CM, Braat DDM, de VRPF, Kremer JAM. Empowering patients undergoing in vitro fertilization by
providing Internet access to medical data. Fertil Steril 2007 Aug;88(2):361-368. [doi: 10.1016/j.fertnstert.2006.11.197][Medline: 17416366]
van DVR, Drossaert CHC, Taal E, Drossaers-Bakker KW, Vonkeman HE, van DLMAFJ. Impact of patient-accessible
electronic medical records in rheumatology: use, satisfaction and effects on empowerment among patients. BMC
Musculoskelet Disord 2014;15:102 [FREE Full text] [doi: 10.1186/1471-2474-15-102] [Medline: 24673997]
Earnest MA, Ross SE, Wittevrongel L, Moore LA, Lin C. Use of a patient-accessible electronic medical record in a practicefor congestive heart failure: patient and physician experiences. J Am Med Inform Assoc 2004;11(5):410-417 [FREE Fulltext] [doi: 10.1197/jamia.M1479] [Medline: 15187074]
Cahill JE, Gilbert MR, Armstrong TS. Personal health records as portal to the electronic medical record. J Neurooncol 2014
Mar;117(1):1-6. [doi: 10.1007/s11060-013-1333-x] [Medline: 24477621]
Steele GC, Miller D, Kuluski K, Cott C. Tying eHealth Tools to Patient Needs: Exploring the Use of eHealth forCommunity-Dwelling Patients With Complex Chronic Disease and Disability. JMIR Res Protoc 2014;3(4):e67 [FREE Fulltext] [doi: 10.2196/resprot.3500] [Medline: 25428028]
Kannry J, Beuria P, Wang E, Nissim J. Personal health records: meaningful use, but for whom? Mt Sinai J Med
2012;79(5):593-602. [doi: 10.1002/msj.21334] [Medline: 22976365]
Chen C, Weider K, Konopka K, Danis M. Incorporation of socioeconomic status indicators into policies for the meaningful
use of electronic health records. J Health Care Poor Underserved 2014 Feb;25(1):1-16 [FREE Full text] [doi:10.1353/hpu.2014.0040] [Medline: 24509007]
Fumagalli LP, Radaelli G, Lettieri E, Bertele' P, Masella C. Patient Empowerment and its neighbours: clarifying theboundaries and their mutual relationships. Health Policy 2015 Mar;119(3):384-394. [doi: 10.1016/j.healthpol.2014.10.017][Medline: 25467286]
Holmström I, Röing M. The relation between patient-centeredness and patient empowerment: a discussion on concepts.
Patient Educ Couns 2010 May;79(2):167-172. [doi: 10.1016/j.pec.2009.08.008] [Medline: 19748203]
Risling T, Martinez J, Young J, Thorp-Froslie N. Evaluating Patient Empowerment in Association With eHealth Technology:
Scoping Review. J Med Internet Res 2017 Sep 29;19(9):e329 [FREE Full text] [doi: 10.2196/jmir.7809] [Medline: 28963090]
eHealth Saskatchewan: CHIP Pilot Program. 2017. URL: https://www.ehealthsask.ca/citizen-engagement/CHIP [accessed
2018-08-20] [WebCite Cache ID 71os9XR8D]

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

Risling et al

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.38
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
29.
30.
31.
32.
33.
34.
35.
36.
37.

38.

39.
40.

41.

Risling et al

eHealth Saskatchewan. 2017. CHIP Pilot Program URL: https://www.ehealthsask.ca/citizen-engagement/CHIP [WebCite
Cache ID 71os9XR8D]
Thorne S, Kirkham SR, MacDonald-Emes J. Interpretive description: a noncategorical qualitative alternative for developing
nursing knowledge. Res Nurs Health 1997 Apr;20(2):169-177. [Medline: 9100747]
Thorne. Interpretive Description: Qualitative Research for Applied Practice (Developing Qualitative Inquiry). New York:
Routledge; 2016.
Thorne S, Con A, McGuinness L, McPherson G, Harris SR. Health care communication issues in multiple sclerosis: aninterpretive description. Qual Health Res 2004 Jan;14(1):5-22. [doi: 10.1177/1049732303259618] [Medline: 14725173]
Funnell MM, Anderson RM, Arnold MS, Barr PA, Donnelly M, Johnson PD, et al. Empowerment: an idea whose time hascome in diabetes education. Diabetes Educ 1991;17(1):37-41. [Medline: 1986902]
Lau DH. Patient empowerment--a patient-centred approach to improve care. Hong Kong Med J 2002 Oct;8(5):372-374[FREE Full text] [Medline: 12376717]
Ammenwerth E, Schnell-Inderst P, Hoerbst A. Patient empowerment by electronic health records: first results of a systematic
review on the benefit of patient portals. Stud Health Technol Inform 2011;165:63-67. [Medline: 21685587]
Ammenwerth E, Schnell-Inderst P, Hoerbst A. The impact of electronic patient portals on patient care: a systematic review
of controlled trials. J Med Internet Res 2012;14(6):e162 [FREE Full text] [doi: 10.2196/jmir.2238] [Medline: 23183044]
Goldzweig CL, Orshansky G, Paige NM, Towfigh AA, Haggstrom DA, Miake-Lye I, et al. Electronic patient portals:
evidence on health outcomes, satisfaction, efficiency, and attitudes: a systematic review. Ann Intern Med 2013 Nov
19;159(10):677-687. [doi: 10.7326/0003-4819-159-10-201311190-00006] [Medline: 24247673]
Jilka SR, Callahan R, Sevdalis N, Mayer EK, Darzi A. “Nothing About Me Without Me”: An Interpretative Review ofPatient Accessible Electronic Health Records. J Med Internet Res 2015;17(6):e161 [FREE Full text] [doi: 10.2196/jmir.4446][Medline: 26123476]
Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: asystematic review. J Med Internet Res 2015;17(2):e44 [FREE Full text] [doi: 10.2196/jmir.3171] [Medline: 25669240]
de LS, Mold F, Sheikh A, Majeed A, Wyatt JC, Quinn T, et al. Patients' online access to their electronic health records andlinked online services: a systematic interpretative review. BMJ Open 2014;4(9):e006021 [FREE Full text] [doi:10.1136/bmjopen-2014-006021] [Medline: 25200561]
Otte-Trojel T, de BA, Rundall TG, van DKJ. How outcomes are achieved through patient portals: a realist review. J Am
Med Inform Assoc 2014 Jul;21(4):751-757. [doi: 10.1136/amiajnl-2013-002501] [Medline: 24503882]

Abbreviations
CHIP: Citizen Health Information Portal
EHR: Electronic health record
ID: interpretive description
WHO: World Health Organization

Edited by G Eysenbach; submitted 24.08.17; peer-reviewed by T Irizarry, L Roa, M Johnson, S Martin; comments to author 01.12.17;
revised version received 16.02.18; accepted 28.06.18; published 10.09.18.
Please cite as:
Risling T, Martinez J, Young J, Thorp-Froslie N
Defining Empowerment and Supporting Engagement Using Patient Views From the Citizen Health Information Portal: Qualitative
Study
JMIR Med Inform 2018;6(3):e43
URL: http://medinform.jmir.org/2018/3/e43/
doi:10.2196/medinform.8828
PMID:30201603©Tracie Risling, Juan Martinez, Jeremy Young, Nancy Thorp-Froslie. Originally published in JMIR Medical Informatics(http://medinform.jmir.org), 10.09.2018. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete
bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license
information must be included.

http://medinform.jmir.org/2018/3/e43/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e43 | p.39
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Original Paper

Emergency Physician Use of the Alberta Netcare Portal, aProvince-Wide Interoperable Electronic Health Record:
Multi-Method Observational Study
Timothy AD Graham1,2, MD; Mark Ballermann1,3, PhD; Eddy Lang4, MD; Michael J Bullard2, MD; Denise Parsons5,
MSc; Gabriella Mercuur5, MD; Pat San Agustin2, MD; Samina Ali5, MD
1

Alberta Health Services, Edmonton, AB, Canada

2

Department of Emergency Medicine, University of Alberta, Edmonton, AB, Canada

3

Department of Critical Care Medicine, University of Alberta, Edmonton, AB, Canada

4

Department of Emergency Medicine, University of Calgary, Calgary, AB, Canada

5

Department of Paediatrics, University of Alberta, Edmonton, AB, Canada

Corresponding Author:
Timothy AD Graham, MD
Alberta Health Services
5th Floor, 44 Capital Boulevard
10044-108 Street
Edmonton, AB, T5J 3S7
CanadaPhone: 1 780 431 2549
Fax: 1 7806139804
Email: tim.graham@ahs.ca

Abstract
Background: The adoption and use of an electronic health record (EHR) can facilitate real-time access to key health informationand support improved outcomes. Many Canadian provinces use interoperable EHRs (iEHRs) to facilitate health informationexchange, but the clinical use and utility of iEHRs has not been well described.
Objective: The aim of this study was to describe the use of a provincial iEHR known as the Alberta Netcare Portal (ANP) in
4 urban Alberta emergency departments. The secondary objectives were to characterize the time spent using the respective
electronic tools and identify the aspects that were perceived as most useful by emergency department physicians.
Methods: In this study, we have included 4 emergency departments, 2 using paper-based ordering (University of Alberta
Hospital [UAH] and Grey Nuns Community Hospital [GNCH]) and 2 using a commercial vendor clinical information system(Peter Lougheed Centre [PLC] and Foothills Medical Centre [FMC]). Structured clinical observations of ANP use and system
audit logs analysis were compared at the 4 sites from October 2014 to March 2016.
Results: Observers followed 142 physicians for a total of 566 hours over 376 occasions. The median percentage of observed
time spent using ANP was 8.5% at UAH (interquartile range, IQR, 3.7%-13.3%), 4.4% at GNCH (IQR 2.4%-4.4%), 4.6% at
FMC (IQR 2.4%-7.6%), and 5.1% at PLC (IQR 3.0%-7.7%). By combining administrative and access audit data, the median
number of ANP screens (ie, results and reports displayed on a screen) accessed per patient visit were 20 at UAH (IQR 6-67), 9
at GNCH (IQR 4-29), 7 at FMC (IQR 2-18), and 5 at PLC (IQR 2-14). When compared with the structured clinical observations,the statistical analysis of screen access data showed that ANP was used more at UAH than the other sites.
Conclusions: This study shows that the iEHR is well utilized at the 4 sites studied, and the usage patterns implied clinical value.
Use of the ANP was highest in a paper-based academic center and lower in the centers using a commercial emergency department
clinical information system. More study about the clinical impacts of using iEHRs in the Canadian context including longer term
impacts on quality of practice and safety are required.
(JMIR Med Inform 2018;6(3):e10184) doi:10.2196/10184

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.40
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

KEYWORDS
ambulatory care facilities; cross-sectional studies; electronic health records; utilization; hospital emergency service; statistics andnumerical data; health information exchange; humans; information dissemination; medical record linkage; program evaluation

Introduction

department. This trend was seen early in the deployment of the
ANP [18].

Background

The ANP was introduced in 2004 in 13 hospitals, 22 publichealth centers, 9 mental health clinics, and a small number ofprivate physician offices within the Edmonton Zone of Alberta
Health Services (formerly known as Capital Health Authority)
and was rapidly adopted by providers in diverse careenvironments [17,18].

The adoption and use of an electronic health record (EHR) can
facilitate real-time access to key health information and support
improved outcomes [1-9]. Canadian hospitals are still primarily
paper based, with only 5.2% of hospitals progressing to stage
five or higher on the 7-point Electronic Medical Record
Adoption Scale [10]. Although most hospital-based care is still
documented on paper, there has been systematic investment
and adoption of digital ancillary systems such as laboratory,
pharmacy, and diagnostic imaging, sponsored by Canada Health
Infoway (Infoway). Infoway is an independent, not-for-profit
organization funded by the federal government, which works
with the provinces and territories to cofund digital healthprojects throughout Canada.
One of the Canadian strategies for digitizing health care has
been the promotion of the interoperable EHR (iEHR), which isequivalent to the health information exchange concept used inother jurisdictions [11,12]. iEHRs are being developed by eachprovince as a part of the wider Canadian initiative to connect
health care nationally. iEHRs are intended to be a longitudinalsummary of key health events and act as a complement topoint-of-service systems such as the electronic medical record(EMR) [13]. iEHRs have been shown to reduce duplicativelaboratory and radiology testing, emergency department costs,
and hospital admissions, with most clinicians also attributing
positive changes in care coordination, communication, andknowledge about patients [14]. In contrast to the iEHR, in
Canada, EMRs tend to be facility centric and refer to an
ambulatory care record primarily linked to one single-care
environment (such as a particular physician’s office) or ahospital-based clinical information system.
In Canada, 12 of the 13 provincial and territorial governments
have implemented iEHRs, but their maturity is quite variable.
To date, only 5 of the 12 have made the 4 planned clinicalcomponents available, including diagnostic images, laboratory
test results, dispensed drugs, clinical reports, and immunizations[15].
Alberta (population 4.2 million) was the first Canadian province
to develop a provincial read-only iEHR known as the Alberta
Netcare Portal (ANP). The ANP iEHR provides read-only accessto province-wide laboratory results, diagnostic imaging (reports
and images), electrocardiograms, textual reports (dictateddischarge summaries, operative reports, some consultations,etc), scanned reports (including the paper emergency department
charts from the Edmonton Zone), dispensed drug information
via the provincial Pharmaceutical Information Network, andcontains data from more clinical domains than any otherprovince [16,17]. ANP has become an important tool forcontinuity of care, and although inpatient care is provided on
paper charts, the information in the ANP has almost entirely
supplanted the need to refer to paper charts in the emergency
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

In 2008, there was a substantial reorganization of health carein Alberta, and multiple health regions were amalgamated into
the first Canadian provincial Health Authority called Alberta
Health Services (AHS). Currently, AHS comprises 5 functionaland regional zones, each with varying degrees of health caredigitization. In acute care facilities, including emergency
departments, the vast majority of the Edmonton Zone uses paperfor documentation and order entry, with digital radiology as
well as electronic patient tracking and laboratory resultsdelivery. In contrast, the Calgary Zone uses an emergency
department and inpatient clinical information system from acommercial vendor (Allscripts), with virtually 100%
computerized provider order entry. The ANP is available as atab and launched from within the Allscripts interface.
Currently, ANP is used in a read-only fashion in 111 hospitals
and more than 650 facilities affiliated within AHS, as well asfamily practice clinics, pharmacies, and other health care entities
in the province. In 2018, over 51,000 physicians, nurses,
pharmacists, and other Alberta health care providers hadrole-based ANP access, with more than 1.9 million patient
records accessed monthly [19]. The ANP’s evolution from a
regional to a province-wide system and the usage patterns andcontrasting information systems between zones in AHS
presented a natural experiment and an excellent opportunity to
study the ANP’s utility and usage in 4 busy urban emergency
departments.

Objective
The primary objective of this study was to compare the use of
an iEHR (health information exchange-type) called ANP andother ancillary electronic systems between emergency
department physicians practicing primarily in paper-based
emergency departments versus clinical information system-based
emergency departments. The secondary objectives were tocharacterize the time spent using the respective electronic tools
and identify the aspects that were perceived as most useful by
emergency department physicians.

Methods
Setting and Participants
We included 4 emergency departments in the study, 2 in the
Edmonton Zone (University of Alberta Hospital [UAH] andGrey Nuns Community Hospital [GNCH]) and 2 in the Calgary
Zone (Peter Lougheed Centre [PLC] and Foothills Medical
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.41
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Centre [FMC]). Data were collected between October 2014 andMarch 2016. The UAH and FMC are comparably large academic
tertiary care facilities and the GNCH and PLC are comparable
community hospitals. The mean number of emergency
department visits annually was 66,003 (UAH), 67,300 (GNCH),
80,466 (FMC), and 79,172 (PLC) during the period of the study.
All physicians providing care in at least one of the 4 study sites
were eligible for inclusion in the direct observation andinterview portions of the study. All uses of the ANP related tothe patients presenting to the emergency department of any
study hospital during the defined study period (October 2014
to March 31, 2016) were considered eligible for inclusion in
the administrative database record review.

Emergency Department Information Systems andTools
The Edmonton Zone and Calgary Zone emergency departments
use different mixes of paper and electronic information systems
to provide care and manage health information (Table 1). Patient
tracking and emergency department lab results in Edmonton
Zone are available on a commercial emergency department
information system, but all order entry and charting is done on
paper. Diagnostic images are available on stand-alone picturearchiving and communication system stations that are notintegrated with the emergency department information system.
Historical health information in Edmonton is usually obtained
via the ANP, which is accessed through a Web browser.
During the study period, care documentation in the Calgary
Zone emergency departments was provided using a paper chart.
Orders were managed using computerized provider order entry
entered into Allscripts, which also provided the relevant
laboratory, diagnostic imaging, and pharmaceutical information
primarily related to Calgary Zone. ANP is available as aparameter-based launch from within the Allscripts interface.
As Allscripts in Calgary contains most relevant patient data, it
was hypothesized that the use of ANP would be much less in
the Calgary zone and primarily related to patient care that
occurred outside the Calgary Zone.

Ethics and Approvals
Research ethics approval was obtained from both the Universityof Alberta (Approval Pro00033249) and the University ofCalgary (Approval REB13-0204) for the observation andinterview portions of the study. The database review including
the ANP access audit data was obtained following guidelines
from provincial legislation that permits the use of data forsystem quality improvement purposes, which was the primary
goal for this data extraction and analysis. The A pRoject Ethics
Community Consensus Initiative screening tool was used as a
guide in this process, and the resulting risk was assessed as
being low.

Structured Clinical Observations
Participants
To prepare for the study, emergency department administrators,
physicians, nurses, and allied health staff were informed about
the study using email notices, visits to departmental meetings,
and posted notices in the emergency departments. Physicians
were approached by observers during their regularly scheduled
shift. Of the 151 physicians approached regarding participation,
146 provided informed consent and 5 declined to participate.
At the initial encounter, physicians provided demographic data,
including their age, sex, payment model, comfort with computers(Likert scale 0-10), comfort interacting with EMR (Likert scale0-10), number of years in practice, and medical credentials.

Procedure
Observations were conducted following previously publishedmethods and using the Work Observation Method by Activity
Timing (WOMBAT) software (WOMBAT 2.0, University of
New South Wales) [20,21]. Research assistants were trained in
the study protocol, use of the tablet-based WOMBAT datacollection tool, and the emergency department patient-tracking
system. The 5 research assistants were trained for at least 12
hours before starting observations. Training also consisted offamiliarization with the data collection tool and data definitions(Multimedia Appendix 1), then completion of an observation
alongside a more experienced observer.

Table 1. Overview of functions available using different applications at the different sites.
Information system function

Application
Emergency Department
Information SystemaaAlberta Netcare Portalb

Sunrise Clinical Manager

Picture Archiving andClinical Information Systemc Communication Systemb

Track patientsYes

Not in real time

Yes

No

View lab results

Yes

Yes

Yes

No

View diagnostic imaging resultsNo

Yes (images and reports)

Yes (images and reports)

Yes (images and reports)

Order medications

No

No

Yes

No

Clinical documentation

No

Text reports (eg, consult
letters)No

No

Handover typed note

Yes

No

Yes

No

Edmonton.

b

Calgary and Edmonton.

c

Calgary.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.42
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
One of the authors (MB) was responsible for training research
assistants and ensured that research assistants achieved
inter-rater agreement scores of at least 90% with respect to thetime spent on tasks and numbers of different tasks recorded
before completing solo observations. Each research assistantconducted observations across multiple sites.
The research assistants approached emergency department
physicians during the earlier hours of their shift, as this was
posited to be when physicians would be more actively involved
in direct clinical care, rather than the end of the shift whensign-overs and charting were being completed. After obtaining
consent, the research assistants observed the physician for 90
min of their clinical shift and recorded demographic and clinicalinformation regarding each patient seen and the time andclinician reason for each clinical access of ANP or otherinformation systems. Time-stamped records including the tasks
being completed, the people who were present, and which
information tools were in use were recorded for later analysis.
Observations were balanced for the time of day (morning 8 am
to 12 pm, afternoon 12 pm to 4 pm, and evening 4 pm to 8 pm)
and time of week (midweek, weekend, Monday, and Friday).
Qualitative field notes supported the capture of additional
contextual information, including the number of learners(medical students and residents) on shift with the physician, thebusyness of physician shift, type of patients seen (ie, chronic,
complex vs ambulatory, pediatric vs geriatric), and type ofpresentation, when possible. Inter-rater reliability was calculated
during training sessions where learning observers were trained
to score the activities alongside more experienced observers,
including the second author. Observers conducted independent
data collection after consistently scoring 90% or higher
agreement on participant time spent on patient care tasks.

Administrative Data Matching and Review
Administrative data for visits to the 4 sites, including the lengthof stay, final disposition (inpatient admission or discharge), anddischarge diagnosis were obtained for October 2014 to March2016. These data are routinely recorded as a part of standard
AHS operations and for auditing provider accesses to patient
records. emergency department encounters were identified froma national system known as the National Ambulatory Care
Reporting System (NACRS) [22], and data matching was
conducted using the available patient identifiers in both the
NACRS and the ANP access audit databases. The number ofscreens viewed on the day of an emergency department
encounter and the subsequent day was retrieved from the accessaudit data. The 2 datasets were matched using a common Unique
Lifetime Identifier, a provincially assigned specific number
used for patient identification purposes. Screen views were
included in the analysis if they happened on either the day ofthe emergency department visit or the subsequent day. A screen
view was defined by a change in the context of the informationthat an emergency department physician accessed on the ANP(Multimedia Appendix 2). For example, after searching for andidentifying a patient, the first screen is a demographics screenwith a Clinical Document Viewer with links to various typesof clinical information. Clicking on a link might provide a single
lab value, a lab flow sheet with multiple values, a textual report,
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

Graham et ala diagnostic image, or an electrocardiogram, each on a different
screen view within the ANP, and each counting as a separate
access.

Analysis
Quantitative analysis was completed using R (3.4.3, The
R-Foundation) and Tableau (10.3). Simple linear regression
was completed to evaluate potential effects of demographic,
practice, and site-related explanatory variables on the percentageof time spent using ANP and other emergency department
information systems tools such as emergency department
information system and picture archiving and communication
system. Additionally, structured physician observation records
included contextual notes to ascertain what information needs
were being met when accessing information tools duringobservations. Differences between sites were evaluated usinga Kruskal-Wallis analysis of variance. The initial study questions
were used to develop the semistructured questionnaire toevaluate what value clinicians found in the tools they used,
including the iEHR, which were evaluated against thequantitative results to support triangulation of the findings.

Results
Structured Clinical Observations
To evaluate the context of emergency department physician useof ANP during clinical work, research assistants recorded
structured clinical observations at the 4 research emergency
departments. Between October 2014 and July 2015, 376
structured, 90-min clinical observations were completed with
142 physician participants (Sites UAH=97, GNCH=94,
FMC=99, and PLC=83). As emergency department physiciansin Calgary provide care across the different sites in each city,
26 participants were observed at more than one site. For over
566 hours of observation, study personnel used the tablet-based
WOMBAT app to record which information resources werebeing used, including the content types they accessed and the
proportion of time emergency department physicians spentaccessing the ANP during each 90-min observation. To evaluate
whether the method was correctly applied, the percentages oftime spent on high-level task categories were calculated andcompared for the 4 emergency departments (Figure 1). The
proportions of time spent on the different categories werecompared and found to be statistically similar across the sites.
The mean proportions of time spent (with 95% CIs) on task
categories in this study were compared with previously
published literature (Figure 1). These percentages of time spent
on different tasks while physician participants provided
emergency department care in Australian hospital wards weresimilar to the values reported in this study than the percentagesof time spent on the high-level task categories by physicians
providing care in Australian hospital wards [20,23].
The median percentage of physician participant time spent using
ANP was 8.5% at UAH (Figure 2; interquartile range, IQR,
3.7%-13.3%), 4.4% at GNCH (IQR 2.4%-4.4%), 4.6% at FMC(IQR 2.4%-7.6%), and 5.1% at PLC (IQR 3.0%-7.7%). The
value at UAH was significantly higher than the other sites(P<.001). The tasks physicians were observed completing mostoften while using ANP are identified in Figure 2.
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.43
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 1. Comparison of proportions of time spent on different tasks in the 4 emergency departments (A) and 2 other studies using the Work Observation
Method by Activity Timing (WOMBAT) method (B). (A) The median and interquartile range of the data for individual observations of physician workat 4 EDs are shown by the symbol and the gray bars. The different task categories are labeled on the x-axis at the top of the figure. University of Alberta
Hospital: filled squares; Grey Nuns Community Hospital: filled circles; Foothill Medical Centre: empty squares; Peter Lougheed Centre: empty circles.
(B) The mean time spent and 95% CIs are reported for this study (+ sign), an Australian study of emergency department physician work (at a registrar
training level comparable with the participants in the current work; x sign), and an Australian study of hospital ward physician work (* sign).

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.44
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 2. Percentage of observed participant time spent using Alberta Netcare Portal (ANP). (A) Each circle represents the percentage of time spentusing ANP for a single 90-min observation session (n=376). The middle black bars represent the median percentage of time spent using ANP and thegray rectangles represent the interquartile range. University of Alberta Hospital site is A, Grey Nuns Community Hospital is B, Foothill Medical Centre
is C, and Peter Lougheed Centre is D. (B) Circles represent the proportion of time spent during individual observation sessions using ANP while
completing the tasks named at the left-hand side.

ANP was used during 20.9% of time spent on indirect patientcare, including reviewing patient information. ANP was usedalongside discussions. Of the total recorded time, 1.5% of therecorded time was spent on Professional Communication. Other
tasks that were completed while physicians used ANP included
Documentation (0.59%), Direct Patient Care (0.55%),
Supervision/Education (0.26%), In Transit (0.06%), andMedication-related tasks (0.01%).
To evaluate other factors that may affect the observed use of
ANP, physician demographic and practice characteristics wererecorded and simple linear regression was used to identify
potential effects on ANP use (Table 2). No statisticallysignificant effects were found.

Administrative Data Matching and Review
Audit logs from the same time period as the structured clinicalobservations (October 2014 to March 31, 2016) were comparedwith the triangulated findings between the 2 methods ofevaluating clinician use. Alongside the structured clinicalobservations, administrative access audit data from ANP werelinked with a routinely generated dataset on emergency
department visits. Patient visits to the 4 study emergency
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

departments between October 2014 and April 2016 wereidentified using administrative data and matched with accessaudit data to derive the number of ANP screens accessed during
3 periods of time (October 2014 through January 2015, July
2015 to September 2015, and February 2016 to April 2016).
An ANP access relating to an emergency department visit was
defined as the use of the ANP after presentation to theemergency department on the same and next calendar day.
During the study period, emergency department physicians at
all sites accessed 1,994,334 lab results, 763,334 diagnostic
imaging results, and 666,222 textual reports. Over 376
observations and 142 emergency department physicians, themean and median number of ANP screens viewed per 90-min
observation in the Edmonton sites were 53 (UAH) and 27(GNCH). In the Calgary sites, the mean number of ANP screensviewed was 15 (FMC) and 13 (PLC; Figure 3). Screen accesses
were calculated at each site, and a median of 20 screens wasaccessed per patient visit at UAH (IQR 6-67), 9 at GNCH (IQR
4-29), 7 at FMC (IQR 2-18), and 5 at PLC (IQR 2-14). When
compared with the structured clinical observations, the statisticalanalysis of screen access data also showed that ANP was used
more at UAH than the other sites.
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.45
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Among the different content types available in ANP, laboratory
and imaging data were accessed more often than transcribedreports, dispensed drug information, or information notcategorized elsewhere (eg, electrocardiograms; Figure 4).

Graham et al
Physicians were observed spending relatively more timereviewing textual reports (mean 4.6%, 95% CI 4.1-5.1) thanlab results (mean 2.6%, 95% CI 2.1-3.1) or diagnostic imaging(mean 1.9%, 95% CI 1.6-2.1; Figure 4).

Table 2. Effect of demographic and practice characteristics on Alberta Netcare Portal (ANP) use as measured by the proportion of time spent.
Demographic and practice characteristics

Sites A and B (n)Sites C and D (n)Age (in years)

.68

<30

2

3

—a

30-39

24

38

—

40-49

23

25

—

50-59

7

10

—

>60

6

4

—

Sex

.06
Female

13

17

—

Male

49

63

—

Credentials.06

Medical Doctorate

41

78

—

Fellow of the Royal College of Physicians of Canada–Emergency
Medicine

27

44

—

American Board of Emergency Medicine

4

5

—

Special Competence in Emergency Medicine

28

29

—

Other (PhD)

3

0

—

Payment schemeaRelationship with observed
ANP usage (P value).47

Fee for service

54

73

—

Salary79—Mixed-compensation model

4

0

—

Years in practice, mean (SD)

13.3 (10)12.9 (9).29

Comfort with computersb, mean (SD)

8.1 (1.6)

8.3 (1.6).35

Comfort with electronic medical recordsb, mean (SD)

7.9 (1.6)

8.1 (1.7).93

Not applicable.

b

Likert scale 0-10; 10=completely comfortable.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.46
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 3. Relative usage of Alberta Netcare Portal across 4 emergency departments from 2014 to 2016 based on access audit data. Each symbol
represents the median weekly count of screens accessed per patient visit to the 4 emergency departments. University of Alberta Hospital: filled squares;
Grey Nuns Community Hospital: filled circles; Foothill Medical Centre: empty squares; Peter Lougheed Centre: empty circles.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.47
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 4. Usage of the different clinical functions in Alberta Netcare Portal based on (A) median counts of screen views from access audit data and(B) observed app usage as measured by the percentage of time spent. The data are shown for each of the 4 sites (A: University of Alberta Hospital; B:
Grey Nuns Community Hospital; C: Foothill Medical Centre; D: Peter Lougheed Centre) for laboratory results (Lab), diagnostic imaging (Imaging),
textual reports (Reports), Pharmaceutical Information Network (PIN; ie, dispensed medication information), and other information (Other). Circles in(B) represent values for individual 90-min observations. Horizontal black bars and gray areas in (B) represent the median and 95% CIs, respectively.

Discussion

resulting in inconsistent and poorly understood clinical impacts[28].

Principal Findings

The triangulation of results in our study, along with thecomparison of the time and motion data with results that havebeen previously reported using the same method in othercountries and care environments, supports the findings.
Clinicians spent a significant amount of time accessing thevarious areas of the ANP and reviewing information relevant
to patient care, consistent with their content. The access audit
data complement these findings. The counts of screen accesses
show information searching behaviors that are temporally linked
to individual patient emergency department visits and implying
they are of important clinical value. Physician perceptions of
the ANP were explored in more detail via semistructured
interviews and are reported elsewhere.

This work triangulates data to evaluate the use of an iEHR
(health information exchange-type) in 4 Canadian emergency
departments. To our knowledge, there have been no prior
Canadian studies describing the use and utility of an iEHR in
the emergency department or any other clinical setting. AcrossCanada, iEHRs are at various stages of implementation andmaturity and have evolved according to provincial and territorial
strategies and priorities [16]. The Alberta Netcare Portal iEHR
is widely used throughout Alberta. A previous study of ANP
use showed 76% of users indicated that it helped provide quality
patient care, whereas 82% felt that it integrated easily into theirclinical workflow [18]. Canadian survey data from 2006 to 2014
demonstrate that in 6 provinces, the Canadian iEHR is perceived
to have positive outcomes in terms of user satisfaction, impact
on quality of care, and impact on productivity [1,15].
In the emergency department setting in other jurisdictions, thesetypes of read-only iEHRs have been shown to be valued byphysicians [2,24,25] and to increase the ability to identify
frequent emergency department users [26]. Usage of iEHRs has
been reported to vary across different practice settings evenwhen the technical configuration is identical [27], potentially
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

In Alberta, it is clear that the ANP has become an integral partof emergency department care and is used extensively, with thehighest observed usage in the complex environment of apaper-based tertiary care academic center. ANP was moreheavily used at the UAH and GNCH (where they document
care primarily on paper and order laboratory testing, diagnostic
imaging, and other interventions either using paper forms or byverbally or telephonically communicating with different
services), compared with sites using a clinical information
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.48
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
system that provided similar information but greater integration
into point-of-care workflows. In contrast to the UAH, the tertiarycare academic site in the Calgary Zone (FMC) had an emergency
department clinical information system, and their relative useof ANP was considerably less—most of the information related
to their local patients was usually embedded within the context
of the emergency department clinical information system.
EHR disuse may result when users and owners do not accrue
the benefits of their use [29]. Here, we found that ANP use wasindependent of the characteristics we evaluated and consistent
with emergency department physicians perceiving that content
available via ANP supported medical decision making. In
particular, information related to the medical management ofpatients and clinical decision support was not available in ANP
and was noted as a potential area of improvement by participantsin the structured clinical observations.
Given that the ANP was utilized less frequently in Calgary
emergency departments where they were able to access muchof the ANP information in their regional clinical information
system, as clinical information system implementations increase
across the country, the role of separate provincial iEHRs needs
further evaluation. Physicians also seek to have clinicaldocumentation, electronic referral, computerized provider order
entry, and clinical decision support all accessible in a single
point-of-care system to support their clinical practice, rather
than logging in and out of multiple systems. Currently, the iEHR
is a read-only app (with some small exceptions) and does notsupport the clinical documentation and ordering needs. Potential
important benefits of clinical information system and electronic
records relate to patient safety, particularly around medication
management and clinical decision support [2,5-7,18], which are
not available in the ANP.

Limitations
Although hybrid paper and computerized practice environments
are common in Canada, our results may or may not generalize
to other settings that do not have the particular mix of thesesystems present in the study emergency departments. The
Alberta context, where an iEHR is available alongside a moretransactional clinical information system in some sites, afforded

Graham et ala unique opportunity to study its use and clinical utility
systematically.
In defining the access of ANP related to an emergency
department visit, the audit logs were not detailed enough to
identify from what part of the hospital the ANP was accessed.
Our ANP accesses may be slight overestimates when a patient
was admitted to the hospital and the ANP was accessed by the
inpatient care team. We believe this possibility to be small
because the majority of patients presenting were not admitted
to an inpatient service, and it can take many hours for an
admitted patient to move from an emergency department to theinpatient bed.
More generally, partially adopted clinical documentation
applications may lead to hybrid environments where some forms
of information are charted on paper and others are charted in aclinical information system [30-32]. This work illuminates theneed to use multiple methods while evaluating the impacts ofdifferent methods of information storage and retrieval in thecontext of fast-paced emergency department care while showing
the clinical utility of a single iEHR accessible across a province.

Conclusions
The current evaluation shows that ANP iEHR is well utilized
at the 4 sites studied, and physicians participating in the study
perceived ANP has a positive impact on knowledge of theirpatients, patient safety, and quality and continuity of care.
Physicians at the paper-based tertiary care hospital utilized ANP
markedly more than those at the clinical information
system-based tertiary care hospital or the 2 community hospitalsites. Physicians working at all 4 sites accessed lab results anddiagnostic imaging more often than textual reports, such asdischarge summaries and operative reports, but spent relatively
more time reviewing textual reports. Physician demographic
and practice characteristics did not predict this common usage
pattern. In its current form, ANP features that could be enhanced
include electronic referral, clinical documentation, andmedication ordering and management. Given the trend ofmoving toward comprehensive clinical information systems to
run hospital systems, the future design, development, andimportance of the iEHR need further evaluation.

Acknowledgments
The authors would like to acknowledge funding support from Alberta Health and Canada Health Infoway. The authors would
also like to acknowledge support in kind from the operational support team for Alberta Netcare Portal and the Chief Medical
Information Office of Alberta Health Services and for permission to access audit log data by the Alberta Health Chief Information
Office. The authors would like to thank Lauren Hirsch and Joyce Tse for their contributions to data collection and Nadia Dow
from the University of Alberta Department of Pediatrics and Tiffany Junghans from the University of Calgary Department ofEmergency Medicine for their coordination of the study. The authors also thank all the emergency department physicians who
participated in the study.

Conflicts of Interest
None declared.

Multimedia Appendix 1
WOMBAT Workflow Definitions.
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.49
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al[PDF File (Adobe PDF File), 89KB - medinform_v6i3e10184_app1.pdf ]Multimedia Appendix 2
Access audit data definitions.
[XLSX File (Microsoft Excel File), 9KB - medinform_v6i3e10184_app2.xlsx ]References
1.
2.

3.

4.
5.

6.

7.

8.
9.
10.
11.
12.

13.

14.

15.

16.
17.
18.
19.
20.

Strating D, MacGregor J, Campbell N. Bridging information “islands”: designing an electronic health record that meetsclinical needs. Healthc Q 2006;9(2):94-98. [Medline: 16640139]
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information
technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006 May 16;144(10):742-752. [Medline:
16702590]
Longhurst CA, Parast L, Sandborg CI, Widen E, Sullivan J, Hahn JS, et al. Decrease in hospital-wide mortality rate after
implementation of a commercially sold computerized physician order entry system. Pediatrics 2010 Jul;126(1):14-21. [doi:10.1542/peds.2009-3271] [Medline: 20439590]
Goldzweig CL, Towfigh A, Maglione M, Shekelle PG. Costs and benefits of health information technology: new trends
from the literature. Health Aff (Millwood) 2009;28(2):w282-w293. [doi: 10.1377/hlthaff.28.2.w282] [Medline: 19174390]
Poon EG, Keohane CA, Yoon CS, Ditmore M, Bane A, Levtzion-Korach O, et al. Effect of bar-code technology on thesafety of medication administration. N Engl J Med 2010 May 6;362(18):1698-1707. [doi: 10.1056/NEJMsa0907115][Medline: 20445181]
Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient
outcomes: a multiple hospital study. Arch Intern Med 2009 Jan 26;169(2):108-114. [doi: 10.1001/archinternmed.2008.520][Medline: 19171805]
Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent
literature shows predominantly positive results. Health Aff (Millwood) 2011 Mar;30(3):464-471 [FREE Full text] [doi:10.1377/hlthaff.2011.0178] [Medline: 21383365]
Jones SS, Adams JL, Schneider EC, Ringel JS, McGlynn EA. Electronic health record adoption and quality improvement
in US hospitals. Am J Manag Care 2010 Dec;16(12 Suppl HIT):SP64-SP71 [FREE Full text] [Medline: 21314225]
Lau F, Kuziemsky C, Price M, Gardner J. A review on systematic reviews of health information system studies. J Am Med
Inform Assoc 2010 Nov;17(6):637-645 [FREE Full text] [doi: 10.1136/jamia.2010.004838] [Medline: 20962125]
Himssanalytics. Health Information Management Systems Society URL: http://www.himssanalytics.org/ [accessed2017-12-08] [WebCite Cache ID 6vYrR1QBJ]
Infocentral.infoway-inforoute. 2017. Standards Centre URL: https://infocentral.infoway-inforoute.ca/en/standards/
standards-centre [accessed 2017-12-08] [WebCite Cache ID 6vYpoEJNM]
Infoway-inforoute. 2016. Digital health blueprint URL: https://www.infoway-inforoute.ca/en/component/edocman/
2944-digital-health-blueprint-enabling-coordinated-collaborative-health-care/download?Itemid=0 [WebCite Cache ID
6vYqGY3So]
Georgiou A, Prgomet M, Paoloni R, Creswick N, Hordern A, Walter S, et al. The effect of computerized provider order
entry systems on clinical care and work processes in emergency departments: a systematic review of the quantitative
literature. Ann Emerg Med 2013 Jun;61(6):644-53.e16. [doi: 10.1016/j.annemergmed.2013.01.028] [Medline: 23548404]
Hersh WR, Totten AM, Eden KB, Devine B, Gorman P, Kassakian SZ, et al. Outcomes from health information exchange:systematic review and future research needs. JMIR Med Inform 2015 Dec 15;3(4):e39 [FREE Full text] [doi:10.2196/medinform.5215] [Medline: 26678413]
Tharmalingam S, Hagens S, Zelmer J. The value of connected health information: perceptions of electronic health record
users in Canada. BMC Med Inform Decis Mak 2016 Jul 16;16:93 [FREE Full text] [doi: 10.1186/s12911-016-0330-3][Medline: 27422571]
Gheorghiu B, Hagens S. Measuring interoperable EHR adoption and maturity: a Canadian example. BMC Med Inform
Decis Mak 2016 Jan 25;16:8 [FREE Full text] [doi: 10.1186/s12911-016-0247-x] [Medline: 26810606]
Albertanetcare. The History of the EHR URL: http://albertanetcare.ca/history.htm [accessed 2017-12-08] [WebCite Cache
ID 6vYt72HAx]
Mayes DC, Mador DR. Evaluating the value and impact of an electronic health record in a complex health system.
ElectronicHealthcare 2010 Mar;8(4):e3-14.
Albertanetcare. EHR Usage Statistics URL: http://albertanetcare.ca/statistics.htm [accessed 2017-12-08] [WebCite Cache
ID 6vYsn6N9x]
Westbrook JI, Ampt A. Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT)
to measure clinicians' patterns of work and communication. Int J Med Inform 2009 Apr;78 Suppl 1:S25-S33. [doi:10.1016/j.ijmedinf.2008.09.003] [Medline: 18951838]

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.50
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
21.

22.
23.
24.
25.

26.

27.

28.

29.
30.

31.
32.

Graham et al

Ballermann MA, Shaw NT, Mayes DC, Gibney RT, Westbrook JI. Validation of the Work Observation Method By Activity
Timing (WOMBAT) method of conducting time-motion observations in critical care settings: an observational study. BMC
Med Inform Decis Mak 2011 May 17;11:32 [FREE Full text] [doi: 10.1186/1472-6947-11-32] [Medline: 21586166]
Cihi. National Ambulatory Care Reporting System Metadata (NACRS) URL: https://www.cihi.ca/en/
national-ambulatory-care-reporting-system-metadata [accessed 2018-05-22] [WebCite Cache ID 6zcDYIkmr]
Westbrook JI, Ampt A, Kearney L, Rob MI. All in a day's work: an observational study to quantify how and with whom
doctors on hospital wards spend their time. Med J Aust 2008 May 5;188(9):506-509. [Medline: 18459920]
Westbrook JI, Coiera E, Dunsmuir WT, Brown BM, Kelk N, Paoloni R, et al. The impact of interruptions on clinical taskcompletion. Qual Saf Health Care 2010 Aug;19(4):284-289. [doi: 10.1136/qshc.2009.039255] [Medline: 20463369]
Johnson KB, Unertl KM, Chen Q, Lorenzi NM, Nian H, Bailey J, et al. Health information exchange usage in emergency
departments and clinics: the who, what, and why. J Am Med Inform Assoc 2011 Sep;18(5):690-697 [FREE Full text] [doi:10.1136/amiajnl-2011-000308] [Medline: 21846788]
Shapiro JS, Johnson SA, Angiollilo J, Fleischman W, Onyile A, Kuperman G. Health information exchange improves
identification of frequent emergency department users. Health Aff (Millwood) 2013 Dec;32(12):2193-2198. [doi:10.1377/hlthaff.2013.0167] [Medline: 24301405]
Lanham HJ, Leykum LK, McDaniel Jr RR. Same organization, same electronic health records (EHRs) system, different
use: exploring the linkage between practice member communication patterns and EHR use patterns in an ambulatory caresetting. J Am Med Inform Assoc 2012;19(3):382-391 [FREE Full text] [doi: 10.1136/amiajnl-2011-000263] [Medline:
21846780]
Healthit.ahrq. 2011. Guide to Reducing Unintended Consequences of Electronic Health Records URL: https://healthit.
ahrq.gov/sites/default/files/docs/publication/guide-to-reducing-unintended-consequences-of-electronic-health-records.pdf[WebCite Cache ID 6vYsn6N9x]
Sittig DF, Singh H. Legal, ethical, and financial dilemmas in electronic health record adoption and use. Pediatrics 2011
Apr;127(4):e1042-e1047 [FREE Full text] [doi: 10.1542/peds.2010-2184] [Medline: 21422090]
Casalino LP, Dunham D, Chin MH, Bielang R, Kistner EO, Karrison TG, et al. Frequency of failure to inform patients ofclinically significant outpatient test results. Arch Intern Med 2009 Jun 22;169(12):1123-1129. [doi:10.1001/archinternmed.2009.130] [Medline: 19546413]
Callen JL, Westbrook JI, Braithwaite J. The effect of physicians' long-term use of CPOE on their test management work
practices. J Am Med Inform Assoc 2006;13(6):643-652 [FREE Full text] [doi: 10.1197/jamia.M2152] [Medline: 16929035]
Stausberg J, Koch D, Ingenerf J, Betzler M. Comparing paper-based with electronic patient records: lessons learned during
a study on diagnosis and procedure codes. J Am Med Inform Assoc 2003;10(5):470-477 [FREE Full text] [doi:10.1197/jamia.M1290] [Medline: 12807808]

Abbreviations
AHS: Alberta Health Services
ANP: Alberta Netcare Portal
EHR: electronic health record
FMC: Foothill Medical Centre
GNCH: Grey Nuns Community HospitaliEHR: interoperable electronic health record
IQR: interquartile range
NACRS: National Ambulatory Care Reporting System
PLC: Peter Lougheed Centre
UAH: University of Alberta Hospital
WOMBAT: Work Observation Method by Activity Timing

Edited by G Eysenbach; submitted 20.02.18; peer-reviewed by F Lau, B Gheorghiu; comments to author 29.03.18; revised version
received 23.05.18; accepted 24.05.18; published 25.09.18.
Please cite as:
Graham TAD, Ballermann M, Lang E, Bullard MJ, Parsons D, Mercuur G, San Agustin P, Ali S
Emergency Physician Use of the Alberta Netcare Portal, a Province-Wide Interoperable Electronic Health Record: Multi-Method
Observational StudyJMIR Med Inform 2018;6(3):e10184
URL: http://medinform.jmir.org/2018/3/e10184/
doi:10.2196/10184
PMID:30274967

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.51
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al©Timothy AD Graham, Mark Ballermann, Eddy Lang, Michael J Bullard, Denise Parsons, Gabriella Mercuur, Pat San Agustin,
Samina Ali. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 25.09.2018. This is an open-accessarticle distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR
Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on
http://medinform.jmir.org/, as well as this copyright and license information must be included.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10184 | p.52
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Original Paper

Task-Data Taxonomy for Health Data Visualizations: Web-Based
Survey With Experts and Older Adults
Sabine Theis1, MSc; Peter Wilhelm Victor Rasche1, MSc; Christina Bröhl1, MSc; Matthias Wille1, Dr Phil; Alexander
Mertens1, Dipl-Inform, Dr Ing, Dr rer medic
Human Factors Engineering and Ergonomics in Healthcare, Chair and Institute of Industrial Engineering and Ergonomics, RWTH Aachen University,
Aachen, Germany

Corresponding Author:
Sabine Theis, MSc
Human Factors Engineering and Ergonomics in Healthcare
Chair and Institute of Industrial Engineering and Ergonomics
RWTH Aachen University
Bergdriesch 27
Aachen, 52062
Germany
Phone: 49 492412418099469
Email: s.theis@iaw.rwth-aachen.de

Abstract
Background: Increasingly, eHealth involves health data visualizations to enable users to better understand their health situation.
Selecting efficient and ergonomic visualizations requires knowledge about the task that the user wants to carry out and the typeof data to be displayed. Taxonomies of abstract tasks and data types bundle this knowledge in a general manner. Task-datataxonomies exist for visualization tasks and data. They also exist for eHealth tasks. However, there is currently no joint task
taxonomy available for health data visualizations incorporating the perspective of the prospective users. One of the most prominent
prospective user groups of eHealth are older adults, but their perspective is rarely considered when constructing tasks lists.
Objective: The aim of this study was to construct a task-data taxonomy for health data visualizations based on the opinion ofolder adults as prospective users of eHealth systems. eHealth experts served as a control group against the bias of lacking
background knowledge. The resulting taxonomy would then be used as an orientation in system requirement analysis and empiricalevaluation and to facilitate a common understanding and language in eHealth data visualization.
Methods: Answers from 98 participants (51 older adults and 47 eHealth experts) given in an online survey were quantitatively
analyzed, compared between groups, and synthesized into a task-data taxonomy for health data visualizations.
Results: Consultation, diagnosis, mentoring, and monitoring were confirmed as relevant abstract tasks in eHealth. Experts andolder adults disagreed on the importance of mentoring (χ24=14.1, P=.002) and monitoring (χ24=22.1, P<.001). The answers to
the open questions validated the findings from the closed questions and added therapy, communication, cooperation, and quality
management to the aforementioned tasks. Here, group differences in normalized code counts were identified for “monitoring”
between the expert group (mean 0.18, SD 0.23) and the group of older adults (mean 0.08, SD 0.15; t96=2431, P=.02).
Time-dependent data was most relevant across all eHealth tasks. Finally, visualization tasks and data types were assigned toeHealth tasks by both experimental groups.
Conclusions: We empirically developed a task-data taxonomy for health data visualizations with prospective users. This provides
a general framework for theoretical concession and for the prioritization of user-centered system design and evaluation. At thesame time, the functionality dimension of the taxonomy for telemedicine—chosen as the basis for the construction of present
taxonomy—was confirmed.
(JMIR Med Inform 2018;6(3):e39) doi:10.2196/medinform.9394
KEYWORDS
classification; data display; computer graphics; task performance and analysis; medicine; telemedicine; user/machine systems;
human factors

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.53
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Introduction
Overview
Health care services are currently undergoing a digitaltransformation that is increasing the amount of clinical andpersonal health data. Data visualizations enable people toanalyze and understand these data to make more informed
decisions and to promote health-improving behavior [1-3].
Information and communication technology (ICT) development
is the driving force behind the digitization of health services.
In the 1990s, digital tools were differentiated from their analog
counterparts with the prefix “e-.” Mail became email andcommerce became e-commerce. Likewise, health became
eHealth. The term describes all health services supported by
ICT [4]. A definition covering all aspects of the term has notbeen achieved to date because it depends on ongoing
technological development and diversity [5]. The major part ofeHealth systems processes data to make it accessible to the user.

Data Visualization
But what does the term data actually mean? Data—as the pluralof the Latium datum—labels “factual information such asmeasurements or statistics used as a basis for reasoning,
discussion, or calculation” [6]. Data results from a measurement[7]. In computer science, “data” is understood asmachine-readable, digital representation of information encoded
into character(s) (strings) following a syntax [8]. In order toabstract the information from data, it must be interpreted in acontext of meaning; therefore, the user must be able to perceive
and understand it [9]. Data visualizations are a way to make useof the effective visual perception channel to exchange
information inherited in data [7]. By assigning graphicalattributes to data, users can grasp data characteristics or identify
new patterns [10-12]. As a graphical representation of data andstatistical concepts, data visualizations particularly support
decision making [13]. Data analysts, scientists, and statisticalexperts have been among the primary users of data visualization
to date [14], but digitization of health services together withdemographic change [15] and the recently observable shift
toward patient empowerment are leading to an increase in thenumber of older adults without special background knowledge
using data visualizations [16-21]. Accordingly, research on thevisualization of health data is increasingly taking into accountthe perspective of older adults for design and evaluation [22-24].

Task Models and Taxonomies
Before developers visualize data, they identify tasks relevant
to users and data relevant to these tasks [25]. This ensures thatvisualization dashboards optimally support users in reaching
their goals. In user-centered development, this is called task
analysis as one method of the requirement analysis [26-28].
Thus, knowledge of visualization tasks is important for theselection or construction of suitable visual representations, atthe same time it supports the empirical visualization evaluation
during the selection of experimental tasks.
Tasks differ in their granularity and degree of abstraction[29,30]. For example, “curing a disease” is a domain task withlow granularity (high-level task), whereas “compare a patient’s
http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

Theis et alheart rate variability data to detect anomalies” describes a
granular domain task (low-level task). Visualization tasks are
determined by the user perspective [31] and numerous models
exist to capture those inferring layers of data visualization tasks
or processes [32-35]. Our work refers to Munzner’s model ofnested layers [36]. Munzner’s nested model describes theprocedure of data visualization design, starting with theinvestigation of domain tasks and data, because users have theirown vocabulary to describe it. Subsequently, the domain
problems have to be translated into abstract visualization tasks
and data types as a vocabulary for data visualization. Data types
in this context are defined by the kind of data to be visualized.
In the third layer of Munzner’s nested model, visual encodings
and interaction methods for data and task abstractions aredeveloped so that corresponding algorithms can be developed
at the innermost level. In this model, the output of one layer isthe input for the subsequent one.
Abstract visualization tasks have often been listed alone or
together with data types in the form of taxonomies [37].
Taxonomies are hierarchical structures originally used to classify
organisms. Later, computer science used them to structureknowledge within knowledge-based systems or forsoftware-testing research [38]. They provide conceptual clarityof a domain and categorize information for increased theoreticalunderstanding. Another advantage is that taxonomies foster
generalizability in empirical research if evaluation considers its
tasks and data types [27,37,39-43]. Taxonomies also allow
precise comparisons across different visualization tools andapplication domains. Work procedures can be analyzed using
a domain-independent language, so that comparative analysesof tasks involving different visualization tools in different
disciplines can be carried out [38,39]. A taxonomy is empiricallybuilt as the hierarchy of the concepts are classified by reason
or measured similarity found in observed variables. A typology,
in contrast, classifies various types that have equalcharacteristics and splits concepts into different types along atleast two dimensions. It does not necessarily rely on empiricalmethods, and elements are less strictly reliant on the hierarchy
as with a taxonomy.
An abstract task typology emerged from Munzner’s [36] nested
model and was developed by Brehmer and Munzner [44]. Theirtypology includes a set of visualization tasks and data types
with different levels of granularity (high level to low level),covering objectives on the “why dimension,” actions on the“how dimension,” and data types on the “what dimension.” We
adopt their definition of data types: kind of data that can bevisualized. The authors state that their typology is relevant fornearly all application domains. Thus, it might be assumed that
it is also relevant for the eHealth domain. Empirical evidence
has yet to be provided and it is one of the objectives of theinvestigation presented in this paper. The typology by Brehmer
and Munzner partly overlaps with the data types from
Shneiderman’s task-by-data-type taxonomy [37]. In a
subsequently published article, Brehmer et al [45] characterizedtask sequences related to the visualization of dimensionallyreduced data. Brehmer et al [46] also encourage detailed
investigations of domain problems and tasks before the actual
design and evaluation.
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.54
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

In the health and eHealth domain, taxonomies of general tasks
have so far been applied to make concepts and their relation
understandable. Furthermore, they are applied to differentiate
ambiguous medical vocabulary [47-51]. For example, Bashshur
et al [47] focused on the differentiation of different terms
describing ICT-mediated health. The authors constructed ataxonomy of telemedicine by differentiating the subdomains
telemedicine, telehealth, eHealth, and mHealth. They
differentiated, as a part of the functionality dimension, theabstract tasks consultation, diagnosis, mentoring, andmonitoring. The described taxonomy was built based on theexpertise of the authors. A user study or literature review was
not undertaken.

the domain of eHealth as possible. Therefore, the purpose ofthis study is to construct a taxonomy of abstract domain andabstract visualization tasks and data types. To the best of ourknowledge, we are the first to investigate the relation between
abstract visualization tasks and data types in the eHealth context
and thus the first to create a taxonomy that has domain relevancebut remains general across different eHealth applications. In
contrast to existing work, we construct the task taxonomy withthe help of prospective eHealth users (older adults), so that it
can foster the understanding of the user, the users’ tasks, andthe users’ domain understanding in order to become a language
among researchers from different domains. In this regard, thestudy will answer the following questions:

Problem Statement

1.

Previous literature illustrated the importance of task analysis
with users for the description, evaluation, and creation of datavisualizations. The problem is that if someone wants to develop
a data visualization system, he or she must first find out whichtasks the users consider relevant by means of user studies.
Abstract visualization tasks as well as data andapplication-specific tasks play a role here. However, if all users
had already been asked for their opinion on relevant tasks anddata, developers could spare this time-consuming step of task
analysis or at least parts of it.
In addition, it is almost impossible for scientists to adhere to
the tasks that are relevant for users during an empiricalevaluation of health data visualizations because this would
require a separate study as a preanalysis of relevant user tasks.
We believe not only developers may profit from using generaltasks relevant to users as input for a more specific requirement
analysis, but also researchers may consider them to selectexperimental tasks so that results from their evaluation becomecomparable and more generalizable across applications [52].
Although an extensive list of task taxonomies for datavisualization exists, they are not suitable to lead developers andscientists to select tasks relevant to users because they are basedon authors’ experience or on literature studies. They lack users’
perspectives. Another problem is that existing health taxonomies
do not consider visualization-specific tasks and data, andtaxonomies or typologies of abstract visualization tasks anddata lack a definition of the domain problem and correspondinguser tasks. Additionally, it remains unclear to what extent
existing visualization task and data type classifications [44,47]
are relevant to prospective eHealth users, who we—given thecontext of demographic change—consider to be older adults.
Older adults are the ones who will use the future systems thatdevelopers can build based on the output of current research
efforts. Furthermore, incidence, prevalence, and mortality arestrongly age dependent. For this reason, the risk of developing
age-dependent chronic diseases or psychological decline isrising. Thus, older adults are more likely to use eHealth systems
than younger people are.

Purpose of the Study
With this study, we want to make a first step toward
generalizable results of user-centered task analysis, so that
results are valuable to as many developers and researchers in
http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

2.

3.

Which abstract eHealth tasks do older adults consider
relevant for eHealth systems?
Which abstract visualization tasks and data types do older
adults consider relevant for medical consultation, diagnosis,
mentoring, and monitoring?
Does the rating from older adults differ from that of eHealthexperts?

Methods
Study Design
We devised a structured cross-sectional study with a nonrandom
sample to collect data from prospective eHealth users (older
adults) and eHealth experts.

Participants
Prospective eHealth users were targeted by focusing on
participants older than 50 years because they are the ones whowill use the future systems that developers can build based on
the output of current research efforts. Furthermore, incidence,prevalence, and mortality are strongly age dependent with risks
rising, for example, for chronic diseases or cognitive andphysical decline [53]. Finally, yet importantly, the handling andperception of technology or relevant tasks is strongly influenced
by the experiences individuals have made with technologicalartifacts during their lives. The so-called technology generations
represent a major influence here [54]. We wanted to focus on
the third group, called the “generation of technology spread”
aged between 53 and 67 years. Thus, a perspective uninfluenced
from existing digital technology could be taken, so thatdevelopers and researchers are able to orient toward the users’
native needs.
We additionally approached eHealth experts to provide evidence
for the validity of the answers from the group of older adults.
Basically, the expert’s answers served as baseline information
to show if and where background knowledge has an impact ornot.

Recruitment
The sampling procedure was nonprobabilistic and purposiveand respondents were selected based on their voluntary
willingness to participate [55,56]. To approach described
experimental groups with differing eHealth backgroundknowledge, different recruitment channels were applied. For
control purposes, the background knowledge was queried withJMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.55
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
only one question instead of with a battery of standard eHealthliteracy questions. This way we could keep the questionnaireas short as possible.
We sent the link to an online survey to eHealth experts fromour existing network in Germany. The survey was presented in
the German language. Then we automatically extracted
additional expert email addresses from the e-health-com
webpage, where readers recommend experts. Editors of thewebsite review the propositions and, if they consider a person
an expert, the website lists them all alphabetically and provides
one profile page per expert containing the name and position
together with a short description, contact information, andaffiliation description. We extracted all email addresses ofexperts automatically from the website by means of a Python
script. We subsequently sent the link to the online questionnaire
to 70 of these experts by email. Of these 70, 24 came fromeHealth industry companies either as chief executive officer of
a company selling eHealth products or as consultant active inthe domain, and 40 came from research institutes working withinformation technology in the health sector. The remainder were
medical experts from various domains or politics.
Older adults were selected by a clickworker platform [57]according to the demographic characteristic of being older than
50 years. Only participants who stated they were 50 years orolder were able to access the survey. The link to the survey wasdisplayed as a task on the website of the platform. At the endof the survey, participants were provided with an individually
generated password. The participants had to provide thepassword to be credited with money to their accounts. We opted
for a fee of €3 for completing the survey, which is relatively
high because it was an abstract, and probably a more difficult
subject, for participants not familiar with it.

Survey Instrument
Data were collected via an online survey. The rationale for theuse of an online questionnaire was that abstract tasks could be
investigated by means of a sample larger than would have beenpossible with observations or qualitative in-depth interviews.
The survey instrument was programmed and made available ona website using SurveyMonkey software [56].
The survey was introduced as a study “improving digital healthcare systems according to user needs” and consisted of fivequestions (for introduction text and survey questions see
Multimedia Appendix 1). All participants were informed about
the duration of the survey, data storage, and the leading
investigator. After an introductory page, individual pages with
one question per screen were displayed. The participant wasable to skip to the next question, but was not able to return tothe previous one. On all survey pages, it was ensured that theuser could see all answer options without the need for scrolling.
The answer options for all questions contained a checkbox withthe label “no answer” (n/a) to keep track if the participant justforgot, or could not, or did not want to provide an answer.
Therefore, answering a question was not mandatory in order to
not frustrate participants and to collect as much information as
the participants wanted to provide.

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

Theis et al
Subsequent to the introductory page, experts and older adults
were asked to list medical tasks that they considered relevant
for health systems (see question #1 in the questionnaire in
Multimedia Appendix 1). This was presented as an open
question to not restrict the participants’ views and to collect asmuch input as possible, while excluding priming effects that
may occur if a list of possible answers was given. The second
question was a closed question asking users to rate the relevanceof consultation, diagnosis, mentoring, and monitoring for
eHealth on a five-point Likert scale (question #2). Subsequently,
participants had to rate the importance of abstract visualization
tasks (“why” dimension) [44] for each task in Bashshur et al’sfunctionality dimension (consultation, diagnosis, mentoring,and monitoring; question #3). Finally, the relevance of datatypes [37,44] for consultation, diagnosis, mentoring, andmonitoring [47] was assessed by means of a checkbox matrix(see question #4) and the background knowledge was assessedby a five-point Likert scale (see question #5). The survey wastested by two independent examiners with regard to wording
and technical functionality.

Data Collection
Data were collected between February 29 and March 14, 2016,
from a sample of eHealth experts, and on November 16, 2016,
from a sample of people older than 50 years without experiencein eHealth. The time interval between the elicitation with experts
and the one with older adults was because of prolonged approval
for using the clickworker portal.
In total, 163 unique individuals visited the website of our
Web-based survey. Identifying individuals was ensured by using
the IP address and cookie function. Of these 163 visitors, 65
never started the survey. In total, 98 visitors participated in thesurvey; the participation rate was 74.4%. The average time spentcompleting the survey was 16 minutes 52.96 seconds.

Analysis
The open-ended answer (see Multimedia Appendix 1, question#1) was first analyzed in terms of the overall word frequencies
with the help of MaxQDA software [58]. Word frequencies
were computed and all occurring words were listed. After theelimination of stop words (eg, in, on, where, why), the resulting
word list was manually scanned for activities and tasks. The
most frequent tasks became an item within a hierarchicaldictionary. The dictionary items were named and structured
referring to Bashshur et al’s [47] functionality dimensions. Eachdimension (consultation, diagnosis, mentoring, or monitoring)
became an item in the dictionary as a child of the root nodeeHealth tasks as soon as it occurred in the word list. Tasks fromthe word frequency list that did not have a “part of” relation
with existing categories were considered the child of the root
node eHealth tasks—and thus a sibling of consultation,
diagnosis, mentoring, or monitoring. Two experienced
qualitative analysts conducted the manual scanning of tasks andthe structuring of the dictionary independently. The two analysts
then discussed differing opinions when they assigned an itemfrom the frequency list to the dictionary or when they sorted
the dictionary and then implemented a common solution. Then,each item (task) in the dictionary contained a list of synonyms
from the word frequency list. For example, the dictionary item
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.56
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS“prevention” contained the words from the frequency list:
prevention, explanatory work, hospital stay, tertiary prevention,
avoidance, and care.
Subsequently, the MaxQDA software automatically coded allwords in the answer texts with the item name from the dictionary
they were assigned to. As a result, the dictionary contained codefrequencies per dictionary item, which added up from lower tohigher structural levels. Consequently, lower levels meant lower
code frequencies. Code frequencies of items on higher levels
were a sum of the item’s own code frequency together with thecode frequencies of all subordinate levels (child items).
For the statistical computation of code count differences among
the two experimental groups, the root level was included up toa maximum of the third level down the hierarchical structure.
For statistical computation, the code frequencies werenormalized with the total number of words the participants gavein their answer. Therefore, for the analysis of the answers on
the closed questions, we used SPSS software, version 22 (IBMCorp, Armonk, NY, USA). To compare answers of eHealthexperts and older adults, t tests for independent samples andchi-square tests were calculated, both at a significance level of.05.

Taxonomy Construction
Our taxonomy for eHealth visualization tasks and data included
the perspective of both experimental groups: the tasks and datatypes that they agreed on and group differences. Individual items
have been ranked from top to bottom, according to task
relevance. The more important an element was, the higher it
was positioned.
Taxonomy construction started with abstract eHealth tasks
resulting from closed question # 2 (see Multimedia Appendix
1) that participants rated as relevant. Tasks resulting from theopen question #1 that were not already referred to by resultsfrom question #2 were then added as siblings. Subsequently,
we added data types from question #3 and the top-ranked
abstract visualization tasks resulting from question #4 to eachof the four abstract eHealth tasks from question #2 (consultation,
diagnosis, mentoring, and monitoring).
Group differences were reflected by the outline of a taxonomy
item. Thick outlines of items illustrated that there were no
significant differences between older adults and eHealth experts,
whereas dotted outlines were significantly more important forexperts and thin-outlined items were significantly more
important for older adults.
Abstract visualization tasks that users most frequently
considered relevant were included in the taxonomy. To
determine the most relevant, we initially ranked all visualization
tasks based on the amount they were considered relevant(“relevance count”). Then we computed the difference between
the relevance counts of consecutive tasks (“relevance countdifference”). The relevance count difference measure served tointensify the differentiation between relevant and nonrelevant
abstract visualization tasks. This reinforcement of the distance
between abstract visualization tasks became necessary in order
to not include too many of them.
http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

Theis et al
All abstract visualization tasks mentioned more frequently thanthe one with the second-biggest relevance count difference toits successor were included in the taxonomy. For example, therelevance of visualization tasks for consultation exhibited thetwo biggest differences between perceive information and search
information (relevance count difference=8) and query
information and lookup information (relevance countdifference=6). In this case, query information and all tasks withhigher total frequency exhibiting no group differences became
part of the taxonomy.

Approval and Informed Consent
The Ethics Committee at RWTH (Rheinisch-Westfälische
Technische Hochschule) Aachen Faculty of Medicine, Germany,
authorized this study and its ethical and legal implications in
its statement EK236/16.

Results
Participants
A total of 98 people participated: 47 eHealth experts and 51
older (≥50 years) adults. The mean age of the eHealth experts
was 42.3 (SD 7.3) years, and the mean age for the older adults
was 55.8 (SD 5.9) years. The eHealth knowledge of the eHealthexperts was comprehensive (8/47, 17%) or very good (39/47,83%), whereas for the older adults it was neutral (15/51, 29%),
low (27/51, 53%), or very low (9/51, 18%).

Relevance of Medical Tasks
The most frequently mentioned eHealth tasks in open-answer
texts were cooperation, consultation, mentoring, monitoring,
documentation, communication, therapy, and quality
management (see Table 1). In contrast to Bashshur et al [47],diagnosis constituted a subtask of therapy. Of all therapy
subtasks, it had the highest frequency, followed by treatment.
Extensions of the original taxonomy could be made concerning
the scope of eHealth tasks, their structure, their validity, andtheir user relevance.
Group differences in the code count were computed on the firstand second level except for the functionality dimension
subconcept therapy, which together with all its child nodes
reached a triple-digit code count. All normalized frequencies
showed a normal distribution. An independent sample t test was
conducted—as the normalized code frequencies were continuous
variables not originating from predefined categories—tocompare the code count of tasks and all child nodes of “therapy”
between older adults and the eHealth experts. There was asignificant difference in the scores for the code frequency ofmonitoring for eHealth experts (mean 0.18, SD 0.23) and older
adults (mean 0.08, SD 0.15; t96=2.43, P=.02). Monitoring wasmore important for experts than for older adults.
The closed question on eHealth task relevance revealed thatacross groups the relevance of eHealth systems for consultationand monitoring was most frequently considered very high. We
received 70 valid answers, of which 51 came from older adults
and 19 from the eHealth expert group (Figure 1, Table 2).

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.57
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Table 1. Task relevance based on code frequencies in open answers in older adults and eHealth experts.
eHealth tasks and subtasks

Word frequencies in older adults, n

Word frequencies in experts, n

Total, N

Cooperation

10

14

24

Consultation (total)

39

66

105

Consultation

25

37

62

Physician-physician

3

18

21

Physician-patient

10

11

21

Physician-pharmacist

1

0

1

Monitoring (total)

42

82

124

Monitoring

23

48

71

Patient condition

0

1

1

Observation

3

0

3

Interpreting data

2

1

3

Data transmission

3

4

7

Data collection

6

8

14

Patient behavior

0

1

1

Medication

0

1

1

Therapy progression

0

4

4

Vital signs

0

13

13

Health condition

1

0

1

Wound surveillance

1

0

1

Identifying saliences

3

1

4

Patient condition

0

1

1

Mentoring (total)

22

21

43

Mentoring

11

11

22

Assistance

5

2

7

Health suggestions

0

2

2

Instructions

6

2

8

Education

0

4

4Documentation (total)

12

11

23

Documentation

6

7

13

Symptoms

101Surgery10

1

Wound documentation

0

2

2

Experience reports

2

0

2

Patient information

2

2

4

Communication (total)

44

52

96

Communication

25

29

54

Data handling/review

7

16

23

Information search

10

3

13

Date arrangement

2

3

5

Billing0

1

1Therapy (total)

98

165

263

Therapy

54

95

149

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.58
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
eHealth tasks and subtasks

Theis et al
Word frequencies in older adults, n

Word frequencies in experts, n

Total, N

Home care2

57

Diagnosis

30

37

67

After treatment

2

4

6

Treatment

6

12

18

Rehabilitation

2

3

5

Prevention

2

9

11

1

3

4

Quality

Figure 1. Mean relevance of individual eHealth tasks according to older adults and eHealth experts. Task relevance rated from 0=very low to 5=veryhigh. Error bars represent 95% CI.

Table 2. Relevance of eHealth tasks in older adults (older) and eHealth experts(expert).
eHealth task

Very low, n (%)

Low, n (%)

Neutral, n (%)

High, n (%)

Very high, n (%)

Total, N

Experts

Older

Experts

Older

Experts

Older

Experts

Older

Experts

Older

Experts

Older

Consultation

0 (0)

3 (6)

1 (5)

1 (5)

2 (11)

8 (16)

7 (37)

19 (37)9 (47)

20 (38)19

51

Diagnosis

0 (0)

2 (4)

2 (11)

7 (14)

4 (21)

7 (14)

9 (47)

21 (41)

4 (21)

14 (28)19

51

Mentoring

0 (0)

2 (4)

2 (11)2 (4)

2 (11)

20 (39)7 (37)

21 (41)8 (42)

3 (6)

19

48

Monitoring

0 (0)

3 (6)

0 (0)

5 (10)0 (0)

11 (22)3 (16)

20 (38)16 (84)12 (24)

19

51

A chi-square test of independence was performed to examine
the relation between relevance counts and user group (older
adults, eHealth experts). The relation between these variables
was highly significant for mentoring (χ24=14.1, P=.002) andmonitoring (χ24=22.1, P<.001). Descriptive values of significant
relevant differences are illustrated in Figures 2 and 3.

Relevance of Abstract Visualization Tasks
The tasks perceive, search, record, present, annotate, and query
information were most important for consultation across thewhole sample. For diagnosis, the priorities were perceive,
discover, search, locate, and identify information. For mentoring,the most relevant abstract visualization tasks were present,
compare, generate, browse, and select information, whereas
http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

monitoring included generate, encode, consume, select, browse,
and compare information (Table 3).

Relevance of Data Types
A chi-square test of goodness-of-fit revealed that data types
relevant to consultation, diagnosis, mentoring, and monitoring
differed significantly between groups for most data types. The
five most relevant data types were included into the taxonomy.
Additionally, the data type relevance for eHealth tasks (Tables
4-7) exhibited few cases in which the relevance frequency
exceeded half the number of valid answers. The most relevant
data types for consultation were quantitative data, nominal data,time-dependent data, points in time, and single values.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.59
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
For diagnosis, time-dependent data, quantitative data, anomalies,
single values, and points in time were most important acrossgroups. Mentoring exhibited time-dependent data, rates ofchange, single values, quantitative data, and points in time as
the most relevant data types.

Theis et al
According to the participants, monitoring required
time-dependent data as the most important data type, followedby temporal patterns, rates of change, and quantitative data, andsingle values. In total, time-dependent and quantitative data
could be numbered among the types with the highest
frequencies.

Figure 2. Relevance of eHealth for mentoring.

Figure 3. Relevance of eHealth for monitoring.

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.60
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Table 3. Abstract visualization tasks relevant for consultation, diagnosis, mentoring, and monitoring in older adults and eHealth experts (N=68).
N

Older adults, n (% from group)eHealth experts, n (% from group)

χ21

P value

Perceive information

53

39 (74)14 (26)0.3.75

Search information

41

27 (66)

14 (34)

4.6.05

Record information41

31 (76)

10 (24)0.1.88Present information40

27 (68)13 (33)

2.9.15

Annotate information40

29 (73)

11 (28)0.3.78

Query information

39

29 (74)10 (26)0.1.88Perceive information47

34 (72)13 (28)0.6.55

Discover information47

34 (72)13 (28)0.6.45

Search information46

33 (2)

13 (28)0.8.37Locate information

43

33 (77)10 (23)0.2.66

Identify information

42

34 (81)8 (19)

2.0.15

Present information36

24 (67)

12 (33)2.8.09Compare information36

27 (75)9 (25)

0.0.99

Generate information

33

23 (70)10 (30)

1.0.33

Browse information

33

22 (67)

11 (33)

2.4.12

Select information

33

22 (67)

11 (33)

2.4.16

Generate information

38

25 (66)

13 (34)3.9.05

Encode information37

33 (89)4 (10)

8.7.01

Consume information35

21 (60)14 (40)8.7.01

Select information35

26 (74)9 (26)

0.2.89

Browse information34

24 (71)

10 (29)0.7.40Compare information34

24 (71)

10 (29)0.7.40Visualization task
Consultation

Diagnosis

Mentoring

Monitoring

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.61
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Table 4. Data types relevant for consultation.
Data types

N

Older adults,

eHealth experts,

Total relevant,n (% from group)n (% from group)n (% from N)

χ21

P value

Quantitative data92

32 (71)15 (32)47(43)

14.1.001

Time dependent95

27 (56)

13 (32)40 (42)8.0.001Single values91

25 (57)13 (28)38 (42)

7.9.01

Points in time92

28 (62)9 (18)

37 (40)

17.7

.001

Nominal data79

19 (59)13 (28)32 (40)

7.9.01

Ordinal data77

16 (53)

13 (28)29 (38)5.1.03Time spans

92

22 (49)7 (15)

29 (32)

12.3

.001

Temporal patterns

90

19 (43)9 (17)

28 (31)

6.6.01Time intervals

91

20 (46)7 (15)

27 (30)

10.2

.001

Anomalies88

19 (46)8 (17)

27 (31)8.9.01Outlier82

14 (30)7 (39)

22 (27)

1.7.21

1-D data

74

9 (33)

12 (26)21 (28)0.5.59Distributions

79

14 (44)7 (15)

21 (27)

8.1.01

Rates of change

91

21 (30)10 (28)31 (34)7.1.01Groups69

8 (15)

8 (17)

16 (23)

3.2.12Time sequences88

14 (34)5 (11)

15 (17)

7.2.01

Synchronizations

82

9 (26)

6 (13)

15 (18)2.3.16Multidimensional data75

5 (18)

10 (21)

15 (20)0.1.78

Clusters

68

6 (29)

5 (11)

11 (16)

3.4.08

2-D data

76

3 (10)

7 (15)

10 (21)

0.3.73

3-D data

74

2 (7)

8 (17)

10 (21)1.4.31

Tree data73

7 (27)

7 (15)

14 (19)

1.6.23

Network data70

3 (13)

7 (15)

10 (14)0.1>.99Graphs78

7 (14)

9 (19)

16 (21)0.1.78

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.62
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Table 5. Data types relevant for diagnosis.
Data types

N

Older adults,

eHealth experts,

Total relevant,n (% from group)n (% from group)n (% from N)

χ21

P valueTime dependent95

37 (77)16 (34)53 (56)

17.8

.001

Quantitative data92

32 (71)16 (34)48 (52)

12.7

.001

Anomalies88

36 (88)12 (25)48 (55)

34.3

.001

Single values

91

32 (73)

13 (28)45 (50)

18.5

.001

Points in time92

32 (71)

11 (23)43 (47)

21.0

.001

Outliers82

24 (69)14 (30)38 (46)

12.1

.001

Time intervals

91

27 (61)

10 (21)37 (41)

15.1

.001

Time spans

92

27 (60)9 (19)

36 (39)

16.1

.001

Nominal data79

24 (75)

11 (23)35 (44)

20.5

.001

Rates of change

91

23 (52)11 (23)34 (37)

8.1.01

Temporal patterns

90

26 (61)7 (15)

33 (37)

20.1

.001

Time sequences

88

24 (59)7 (15)

31 (37)

18.3

.001

Ordinal data77

18 (60)11 (23)29 (38)10.5.01

2-D data

76

17 (59)12 (26)29 (38)8.3.01

1-D data74

17 (63)

11 (23)28 (38)

11.0

.001

Graphs78

19 (61)9 (19)

28 (36)

14.4

.001

3-D data

74

15 (56)

11 (23)26 (35)7.8.01

Distributions97

17 (53)9 (19)

26 (33)10.0.01Multidimensional data75

12 (43)

12 (26)24 (32)2.4.13Groups96

15 (68)8 (17)

23 (33)

17.7

.001

Clusters86

13 (62)9 (19)

22 (32)

12.1

.001

Synchronizations82

13 (37)5 (11)

18 (22)8.2.01

Net data

70

8 (35)

9 (19)

17 (24)

3.0.23

Tree data73

10 (39)6 (13)

16 (22)

6.5.02

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.63
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Table 6. Data types relevant for mentoring.
Data types

N

Older adults,

eHealth experts,

Total relevant,n (% from group)n (% from group)n (% from N)

χ21

P valueTime dependent95

18 (38)13 (28)31 (33)

1.1.38

Rates of change

91

19 (43)

12 (26)31 (34)3.1.08Single values

91

23 (52)8 (17)

31 (34)

12.6

.001

Quantitative data92

17 (38)12 (26)29 (32)

1.6.26

Points in time92

18 (40)11 (23)29 (32)

2.9.12Time spans

92

19 (42)9 (19)

28 (32)

5.8.02

Temporal patterns

90

17 (40)11 (23)28 (31)

2.0.12Anomalies88

18 (40)9 (19)

27 (31)

6.3.02

Nominal data79

13 (41)

13 (28)26 (33)

1.5.33Time intervals

91

14 (32)10 (21)24 (36)

1.3.34Time sequences

88

16 (39)8 (17)

24 (27)

5.3.03

Graphs78

15 (48)8 (17)

23 (30)8.8.01

Ordinal data77

9 (30)

13 (28)22 (29)0.1.99

1-D data

74

12 (44)9 (19)

21 (29)

5.4.03

Clusters68

10 (48)9 (19)

19 (28)5.

.02

2-D data

76

10 (35)8 (17)

18 (24)3.0.10Distributions

79

10 (31)8 (17)

18 (23)2.2.18

3-D data

74

8 (30)

9 (19)

17 (23)

1.1.39

Synchronizations82

12 (34)5 (11)

17 (21)

6.8.01Multidimensional data79

10 (36)7 (15)

17 (21)

4.3.05

Outlier82

10 (29)7 (15)

17 (21)

2.3.02Tree data73

10 (39)6 (13)

16 (22)6.5.02Groups69

8 (36)

8 (17)

16 (23)3.2.12

Net data

70

8 (35)

7 (15)

15 (21)

3.6.07

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.64
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Table 7. Data types relevant for monitoring.
Data types

N

Older adults,

eHealth experts,

Total relevant,n (% from group)n (% from group)n (% from N)

χ21

P valueTime dependent95

22 (46)15 (32)37 (40)1.9.21

Temporal patterns

90

22 (51)

13 (28)35 (39)5.2.03

Rates of change

91

21 (48)12 (26)33 (36)

4.8.03

Quantitative data92

14 (31)

17 (26)31 (34)

0.3.66

Points in time92

20 (44)11 (23)31 (34)4.6.05

Single values91

18 (41)

13 (28)31 (34)

1.8.19Time spans

92

19 (43)

11 (23)30 (33)3.7.08Graphs78

20 (65)10 (21)30 (39)

14.6

.001

Synchronizations82

21 (60)8 (17)

29 (35)

16.2

.001

Multidimensional data75

16 (57)13 (28)29 (39)6.4.02Time intervals

91

16 (36)

12 (26)28 (31)

1.3.36

2-D data

76

18 (46)9 (19)

27 (36)

14.4

<.001

Time sequences88

17 (42)10 (21)27 (31)

4.5.06Outliers82

14 (40)12 (26)26 (32)

1.9.23

Anomalies88

17 (42)9 (19)

26 (30)5.2.03

3-D data

74

14 (53)

11 (23)25 (34)

6.2.02

Distributions

79

16 (50)9 (19)

25 (32)

8.4.01

Nominal data79

15 (47)9 (19)

24 (30)

6.9.01

Ordinal data77

9 (30)

12 (26)21 (27)0.2.79Groups69

11 (50)10 (21)21 (30)5.8.02

1-D data

74

10 (37)10 (21)20 (27)

2.2.18

Clusters68

10 (48)9 (19)

19 (28)5.8.02

Tree data73

11 (42)7 15

18 (25)6.8.01

Net data

70

9 (39)

8 (17)

17 (24)

4.1.07

Task-Data Taxonomy for eHealth Visualizations
The task-data taxonomy for eHealth visualizations wasconstructed as described in the Methods section of our paper.
It shows which health tasks are important for them and which
abstract visualization tasks and data types are relevant for theabstract health tasks monitoring,” consultation, diagnosis, andhttp://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

mentoring. Group differences within the taxonomy are marked
with different outline characteristics of the taxonomy item,
which gives it higher meaning (dotted line=experts, thinline=older adults, thick line=no difference). It is striking that
all relevant abstract visualization tasks were considered relevant
by both groups, so there were no significant differences in
relevance (see Figure 4).

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.65
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Figure 4. The eHealth visualization task-data taxonomy.

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.66
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al

Discussion

discrepancy between groups regarding the importance of thetask monitoring was replicated as well.

Principal Findings

Against the background of current work on the development of
eHealth applications [9,62-66], we would have expected
monitoring to be the most relevant eHealth task. The results ofcode count frequencies do not match this expectation. Because
the results of previous studies on the investigation ofhealth-related information need are consistent with the fact that,for adults older than 50 years, diagnosis is the most important
information during the maintenance and administration of theirpersonal health [67], it can be assumed that older adults regard
the relevance of individual eHealth tasks less from a technology
perspective. Tasks that are important for personal health haveincreased importance for older adults.

This section offers a discussion and interpretation of the results
regarding the task analysis of eHealth and visualization tasks
and the corresponding data types across the two experimentalgroups: eHealth experts and older adults. We additionallyelaborate on the limitations of our findings and describe futurework.
The eHealth experts’ answer texts led to a total of 244 codes,
whereas 155 codes could be derived from the older adults’answer texts. Here, therapy was most frequently mentioned
across the whole sample with a number of 263 counts including
all subtasks (see Table 1), followed by monitoring (n=124),
consultation (n=105), communication (n=96), mentoring (n=43),
documentation (n=23), and quality management (n=4).
Monitoring was seen differently across user groups: it wassignificantly more important to the experts than to the older
adults. Diagnosis was found to be the most frequently mentioned
subtask of therapy followed by treatment, prevention, homecare, aftertreatment, and rehabilitation. The tasks at the second
level were cited less frequently. The therapeutic tasks users
considered most important were diagnosis and treatment. The
former is important for both groups, whereas medical or eHealthexperts cited treatment and prevention twice or more frequently.
Collecting data as well as monitoring of vital data were the mostcommonly mentioned subtasks of monitoring in the participants’opinions. Similar to the task at the first level (monitoring), there
is a clear group difference with a focus on the maximum in theexpert group.
It appears that code frequencies are relatively low compared to
the whole sample size. This can be explained by the short,keyword-like answers most participants gave. For example, thesample group of 98 mentioned monitoring only 61 times.
Considering that each code count cannot even be exclusively
assigned to one person, results from a starting point for
taxonomy construction requires future iterative improvementwith a larger sample size as well as a validation of thehierarchical arrangement of individual elements [59,60].
Results of the open answers confirm the relevance of thefunctionality dimension within the taxonomy of telemedicine
and that given task classification could be extended by the tasks
therapy, cooperation, documentation, communication, andquality management. Results regarding confirmation of thefunctionality dimension of the taxonomy of telemedicine are inline with the results of our previous work [61].
The abstract eHealth tasks of the functional dimensions formed
the root nodes of our taxonomy by their later assignment toabstract visualization tasks and data types; therefore, the validityof the analysis of the open and uninfluenced responses wasvalidated by directly querying their importance with five-point
Likert scales. The analysis of those closed questions on therelevance of the tasks consultation, diagnosis, mentoring, andmonitoring supported results from the qualitative content
analysis of open questions. Here, both user groups considered
monitoring and diagnosis the most important eHealth tasks. The
http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

The background knowledge of older adults regarding thetechnical possibilities of eHealth systems differs from that ofeHealth experts. Conventional constant monitoring or medical
control has been less important to laypeople because it might
be unclear to them that when it comes to continuous monitoringof sensor data, technical systems are often more accurate andstable at monitoring patients than medical personnel. The mental
model that seems to influence the answer—even if the term was
explained at the beginning of the survey—is more strongly
characterized by health-relevant tasks users know from theireveryday life, where the extensive introduction of digitalmonitoring systems is still pending in Germany.
At first sight, one might suspect this is a problem for the utilityof the developed task-data taxonomy. However, this is only thecase if one assumes that our taxonomy should precisely represent
the tasks currently present in systems. However, the aim of thetaxonomy is—as described at the outset—an increase in theuser-centricity of future systems. For our taxonomy, it is notimportant which tasks and data actually exist, but which are
relevant for prospective users, so that systems developed basedon presented taxonomy have the greatest possible value.
However, users’ perceptions of the relevance of individual tasks
and data types are of great importance.
The question on the relevance of abstract visualization tasks
was not answered by nearly a third of the participants (30/98).
Whether a lack of knowledge or a lack of motivation is
responsible cannot be determined on the basis of the data.
Because 75% of older adults and only 25% of experts answered
the question, despite experts having higher backgroundknowledge, motivation seems to be more likely an influencing
factor here.
We also assume that the eHealth systems including suchvisualizations are not available to some participants.
Therefore—as in the case of the abstract eHealth tasks—theresults identify potential areas where data visualizations could
enable experts or patients to be supported in the corresponding
medical task.
Our ranking of general eHealth tasks supports the general
understanding of the application context of eHealth and eHealthvisualizations from the perspective of prospective users (older
adults). Visualizations that support those general domain tasks
are expected to have a stronger impact. The intention here is
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.67
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
not to invite visualization researchers to contribute designs to
the eHealth domain, but to identify potential for the applicationof visualizations within eHealth systems, an aspect that hasoften been overlooked.

Transfer of Knowledge
The presented results add to the increasing number of papers
that target hierarchical task structures to establish a common
vocabulary and understanding of visualization tasks and data
[68,69]. This work goes beyond that by considering the context
of eHealth including the perspective of the prospective user andsynthesizing their input in the form of eHealth task-datataxonomy. In this way, eHealth system developers andresearchers can use it as an orientation during requirement
analysis or as a guideline for the definition of experimental tasks
in visualization evaluation experiments.

Limitations
We consider the described eHealth task-data taxonomy asprovisional and subject to validation in the field. In addition,
we only tracked the subjectively perceived knowledge about
eHealth systems, so participants might lack familiarity with
abstract data types or task-data taxonomies or they may not befamiliar with online surveys and interactive Web tools such as
those used for our Web survey. Thus, we are not able to quantify
participants’ familiarity with concepts mentioned in this study
and this may have influenced our findings. Familiarity with
abstract data types and visualization tasks and styles common
to the survey website would have likely reduced some of thebarriers participants might have experienced.

Theis et al
Furthermore, as with subjective methods in general, results arelimited in a way that they reflect the perspective and mental
model of the participants together with their experiences. But
observations will have the drawback that achievable sample
sizes are much smaller, so that the results are hardly
generalizable to the whole eHealth domain.
Additional limitations of our study lie in the selective sample
caused by using an online questionnaire. People who are familiar
with technology are more likely to answer the questionnairethan people who are not. Additionally, the older adults were
paid, whereas the experts were not. This leads to different
motivations between the two groups, which could be an
influencing variable. This might have been the reason why the
numbers of completed answers varied in the expert group overthe length of the questionnaire (more were answered at the
beginning than at the end).

Conclusion
We successfully constructed a task-data taxonomy for eHealthdata visualizations by providing a general description of tasks
and data useful for health data visualizations. We have shownthat semantic approaches [26] are feasible to generally performtask analysis. Furthermore, the results empirically validated andranked Brehmer and Munzner’s [44] typology of abstract
visualization tasks, as well as the functionality dimension of
Bashshur et al’s [47] taxonomy of telemedicine. Time-dependent
data and searching for information within visualizations ofmonitoring data had the highest relevance across user groups.

Acknowledgments
This publication is part of the research project “TECH4AGE,” financed by the Federal Ministry of Education and Research(BMBF, under grant no: 16SV7111) and promoted by VDI/VDE Innovation + Technik GmbH.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Text survey introduction and questionnaire.
[PDF File (Adobe PDF File), 355KB - medinform_v6i3e39_app1.pdf ]References
1.
2.
3.

4.
5.
6.

Shneiderman B, Plaisant C, Hesse BW. Improving healthcare with interactive visualization. Computer 2013 May;46(5):58-66.
[doi: 10.1109/MC.2013.38]
Faisal S, Blandford A, Potts HW. Making sense of personal health information: challenges for information visualization.
Health Informatics J 2013 Sep;19(3):198-217. [doi: 10.1177/1460458212465213] [Medline: 23981395]
Hill B, Proulx J, Zeng-Treitler Q. Exploring the use of large clinical data to inform patients for shared decision making.
Medinfo 2013 2013;192(Studies in Health Technology and Informatics):851-855 [FREE Full text] [doi:10.3233/978-1-61499-289-9-851] [Medline: 23920678]
World Health Organization. eHealth at WHO: resolutions and deliberations on eHealth URL: http://www.who.int/ehealth/about/en/ [accessed 2018-03-07] [WebCite Cache ID 6xk8lgv4C]
Oh H, Rizo C, Enkin M, Jadad A. What is eHealth (3): a systematic review of published definitions. J Med Internet Res
2005 Feb 24;7(1):e1 [FREE Full text] [doi: 10.2196/jmir.7.1.e1] [Medline: 15829471]
International Organization for Standardization. 2015 May. ISO/IEC 2382:2015: Information technology-vocabulary URL:
https://www.iso.org/standard/63598.html [accessed 2018-06-08] [WebCite Cache ID 701trcEMA]

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.68
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
7.
8.
9.

10.

11.

12.

13.
14.

15.

16.

17.
18.

19.

20.
21.
22.
23.

24.
25.
26.

27.
28.

29.

Wilkinson L. The Grammar of Graphics. 2nd Edition. New York: Springer Verlag; 2005.
Gumm H, Sommer M, Hesse W. Einführung in die Informatik. 10th Edition. München: Oldenbourg; 2013.
Edbrooke-Childs J, Smith J, Rees J, Edridge C, Calderon A, Saunders F, et al. An app to help young people self-manage
when feeling overwhelmed (ReZone): protocol of a cluster randomized controlled trial. JMIR Research Protocols 2017
Nov 03;6(11):e213 [FREE Full text] [doi: 10.2196/resprot.7019] [Medline: 29101094]
Piro NE, Piro LK, Kassubek J, Blechschmidt-Trapp RA. Analysis and visualization of 3D motion data for UPDRS ratingof patients with Parkinson's disease. Sensors (Basel) 2016 Jun 21;16(6):1 [FREE Full text] [doi: 10.3390/s16060930][Medline: 27338400]
Jusufi I, Nyholm D, Memedi M. Visualization of spiral drawing data of patients with Parkinson's disease. 2014 Presented
at: 18th International Conference on Information Visualisation (IV); Jul 16-18, 2014; Paris, France p. 346-350 URL: https://ieeexplore.ieee.org/abstract/document/6902928/ [doi: 10.1109/IV.2014.31]
Serrano JA, Thoms A, Weber P. Patients initiated timeline marking of events in Parkinson’s disease: visualization of timecorrelation between patients marked events and acquired data from sensors. In: Schmorrow DD, Fidopiastis CM, editors.
Foundations of Augmented Cognition. Advancing Human Performance and Decision-Making through Adaptive Systems.
Cham: Springer; 2014:325-334.
Laramee RS, Ware C. Rivalry and interference with a head-mounted display. ACM Trans Comput-Hum Interact
2002;9(3):238-251. [doi: 10.1145/568513.568516]
Goren-Bar D, Shahar Y, Galperin-Aizenberg M, Boaz D, Tahan G. KNAVE II: The definition and implementation of an
intelligent tool for visualization and exploration of time-oriented clinical data. In: Proceedings of the Working Conference
on Advanced Visual Interfaces. 2004 Presented at: AVI '04 Working Conference on Advanced Visual Interfaces; May
25-28, 2004; Gallipoli, Italy p. 171-174. [doi: 10.1145/989863.989889]
Buck H, Kistler E, Mendius HG. How will demographic change impact the labour market and the world of work? In:
Demographic Change in the World of Work: Opportunities For an Innovative Approach to Work—A German Point ofView. Stuttgart: Bundesministerium für Bildung und Forschung; 2002:15-35.
Stewart CD, Hanson VL, Nind TJ. Assisting older adults in assessing the reliability of health-related websites. In: CHI '14
Extended Abstracts on Human Factors in Computing Systems. 2014 Presented at: ACM CHI Conference on Human Factors
in Computing Systems; Apr 26-May 1, 2014; Toronto, ON p. 2611-2616. [doi: 10.1145/2559206.2581243]
Le T, Reeder B, Chung J, Thompson H, Demiris G. Design of smart home sensor visualizations for older adults. Technol
Health Care 2014;22(4):657-666. [doi: 10.3233/THC-140839] [Medline: 25267608]
Latulipe C, Gatto A, Nguyen HT, Miller DP, Quandt SA, Bertoni AG, et al. Design considerations for patient portal adoption
by low-income, older adults. In: Proc SIGCHI Conf Hum Factor Comput Syst. 2015 Apr Presented at: 33rd Annual ACM
Conference on Human Factors in Computing Systems; Apr 18-23, 2015; Seoul, Korea p. 3859-3868 URL: http://europepmc.
org/abstract/MED/27077140 [doi: 10.1145/2702123.2702392]
Thompson HJ, Demiris G, Rue T, Shatil E, Wilamowska K, Zaslavsky O, et al. A Holistic approach to assess older adults'
wellness using e-health technologies. Telemed J E Health 2011 Dec;17(10):794-800 [FREE Full text] [doi:10.1089/tmj.2011.0059] [Medline: 22011052]
Huh J, Le T, Reeder B, Thompson HJ, Demiris G. Perspectives on wellness self-monitoring tools for older adults. Int J
Med Inform 2013 Nov;82(11):1092-1103 [FREE Full text] [doi: 10.1016/j.ijmedinf.2013.08.009] [Medline: 24041452]
Kopanitsa G, Hildebrand C, Stausberg J, Englmeier KH. Visualization of medical data based on EHR standards. Methods
Inf Med 2013;52(1):43-50. [doi: 10.3414/ME12-01-0016] [Medline: 23223709]
Le T, Thompson HJ, Demiris G. A comparison of health visualization evaluation techniques with older adults. IEEE Comput
Graph Appl 2016;36(4):67-77. [doi: 10.1109/MCG.2015.93] [Medline: 26415162]
Harte RP, Glynn LG, Broderick BJ, Rodriguez-Molinero A, Baker PM, McGuiness B, et al. Human centred design
considerations for connected health devices for the older adult. J Pers Med 2014 Jun 04;4(2):245-281 [FREE Full text][doi: 10.3390/jpm4020245] [Medline: 25563225]
Le T, Reeder B, Yoo D, Aziz R, Thompson HJ, Demiris G. An evaluation of wellness assessment visualizations for older
adults. Telemed J E Health 2015 Jan;21(1):9-15 [FREE Full text] [doi: 10.1089/tmj.2014.0012] [Medline: 25401414]
Nardi BA. Activity Theory and Human-Computer Interaction. Cambridge, MA: Massachusetts Institute of Technology;
1995.
Theis S, Rasche P, Bröhl C, Wille M, Mertens A. User-driven semantic classification for the analysis of abstract health andvisualization tasks. In: Duffy V, editor. Digital Human Modeling. Applications in Health, Safety, Ergonomics, and Risk
Management: Health and Safety. DHM 2017. Lecture Notes in Computer Science, vol 10287. Cham: Springer; 2017:297-305.
Ahn J, Plaisant C, Shneiderman B. A task taxonomy for network evolution analysis. IEEE Trans Vis Comput Graph 2014
Mar;20(3):365-376. [doi: 10.1109/TVCG.2013.238] [Medline: 24434218]
International Organization for Standardization. 2011 Jan. ISO 9421-210 Ergonomics of human-system interaction-Part 210
human-centred design for interactive systems URL: https://www.iso.org/standard/52075.html [accessed 2018-06-08][WebCite Cache ID 701eZsJmr]
Rind A, Aigner W, Wagner M, Miksch S, Lammarsch T. User tasks for evaluation: untangling the terminology throughout
visualization design and development. In: BELIV '14 Proceedings of the Fifth Workshop on Beyond Time and Errors:

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

Theis et al

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.69
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

30.
31.
32.

33.

34.
35.
36.
37.
38.
39.
40.

41.
42.

43.
44.
45.

46.

47.
48.

49.
50.
51.
52.

Novel Evaluation Methods for Visualization. 2014 Presented at: BELIV '14 Fifth Workshop on Beyond Time and Errors:
Novel Evaluation Methods for Visualization; Nov 10, 2014; Paris, France p. 9-15. [doi: 10.1145/2669557.2669568]
Schulz H, Nocke T, Heitzler M, Schumann H. A design space of visualization tasks. IEEE Trans Vis Comput Graph 2013
Dec;19(12):2366-2375. [doi: 10.1109/TVCG.2013.120] [Medline: 24051803]
Miksch S, Aigner W. A matter of time: applying a data-users-tasks design triangle to visual analytics of time-oriented data.
Comput Graph 2014;38:286-290. [doi: 10.1016/j.cag.2013.11.002]
Amar R, Stasko J. BEST PAPER: A knowledge task-based framework for design and evaluation of information visualizations.
In: IEEE Symposium on Information Visualization 2004. 2004 Presented at: INFOVIS 2004: IEEE Symposium on Information
Visualization; Oct 10-12, 2004; Austin, TX p. 143-150. [doi: 10.1109/INFVIS.2004.10]
Amar R, Eagan J, Stasko J. Low-level components of analytic activity in information visualization. In: Proceedings of the2005 IEEE Symposium on Information Visualization. 2005 Presented at: INFOVIS 2005: IEEE Symposium on Information
Visualization; Oct 23-25, 2005; Minneapolis, MN p. 111-117. [doi: 10.1109/INFOVIS.2005.24]
Albers D, Correll M, Gleicher M. Task-driven evaluation of aggregation in time series visualization. Proc SIGCHI Conf
Hum Factor Comput Syst 2014;2014:551-560 [FREE Full text] [doi: 10.1145/2556288.2557200] [Medline: 25343147]
Rind A, Aigner W, Wagner M, Miksch S, Lammarsch T. Task Cube: a three-dimensional conceptual space of user tasks
in visualization design and evaluation. Inform Visual 2016 Jul 25;15(4):288-300. [doi: 10.1177/1473871615621602]
Munzner T. A nested model for visualization design and validation. IEEE Trans Vis Comput Graph 2009;15(6):921-928.
[doi: 10.1109/TVCG.2009.111] [Medline: 19834155]
Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. 1996 Presented at: IEEE
Symposium on Visual Languages; Sep 3-6, 1996; Boulder, CO p. 336-343.
Vegas S, Juristo N, Basili V. Maturing software engineering knowledge through classifications: a case study on unit testing
techniques. IIEEE Trans Software Eng 2009 Jul;35(4):551-565. [doi: 10.1109/TSE.2009.13]
Ellis G, Dix A. A taxonomy of clutter reduction for information visualisation. IEEE Trans Vis Comput Graph
2007;13(6):1216-1223. [doi: 10.1109/TVCG.2007.70535] [Medline: 17968067]
Valiati E, Pimenta M, Freitas C. A taxonomy of tasks for guiding the evaluation of multidimensional visualizations. In:
Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information
Visualization. 2006 Presented at: BELIV '06 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for
Information Visualization; May 23, 2006; Venice, Italy p. 1-6. [doi: 10.1145/1168149.1168169]
Ward MO. A taxonomy of glyph placement strategies for multidimensional data visualization. Inform Visual 2016 Nov
30;1(3-4):194-210. [doi: 10.1057/palgrave.ivs.9500025]
Paul CL, Whitley K. A taxonomy of cyber awareness questions for the user-centered design of cyber situation awareness.
In: Marinos, L, Askoxylakis I, editors. Human Aspects of Information Security, Privacy, and Trust. Berlin: Springer Berlin
Heidelberg; 2013:145-154.
Tory M, Möller T. Human factors in visualization research. IEEE Trans Vis Comput Graph 2004;10(1):72-84. [doi:10.1109/TVCG.2004.1260759] [Medline: 15382699]
Brehmer M, Munzner T. A multi-level typology of abstract visualization tasks. IEEE Trans Vis Comput Graph 2013
Dec;19(12):2376-2385. [doi: 10.1109/TVCG.2013.124] [Medline: 24051804]
Brehmer M, Sedlmair M, Ingram S, Munzner T. Visualizing dimensionally-reduced data: interviews with analysts and acharacterization of task sequences. 2014 Nov 10 Presented at: Fifth Workshop on Beyond Time and Errors: Novel Evaluation
Methods for Visualization; 2014; Paris, France p. 1-8. [doi: 10.1145/2669557.2669559]
Brehmer M, Carpendale S, Lee B, Tory M. Pre-design empiricism for information visualizationcenarios, methods,challenges.
2014 Presented at: Fifth Workshop on Beyond TimeErrors: Novel Evaluation Methods for Visualization; Nov 10, 2014;
Paris, France p. 147-151. [doi: 10.1145/2669557.2669564]
Bashshur R, Shannon G, Krupinski E, Grigsby J. The taxonomy of telemedicine. Telemed J E Health 2011;17(6):484-494.
[doi: 10.1089/tmj.2011.0103] [Medline: 21718114]
Tulu B, Chatterjee S, Laxminarayan S. A taxonomy of telemedicine efforts with respect to applications, infrastructure,delivery tools, type of setting and purpose. 2005 Presented at: 38th Annual Hawaii International Conference on System
Sciences, 2005; Jan 6, 2005; Big Island, HI p. 147b. [doi: 10.1109/HICSS.2005.56]
Ingenerf J. Telemedicine and terminology: different needs of context information. IEEE Trans Inf Technol Biomed 1999
Jun;3(2):92-100. [Medline: 10719490]
Chan CV, Matthews LA, Kaufman DR. A taxonomy characterizing complexity of consumer eHealth Literacy. AMIA Annu
Symp Proc 2009 Nov 14;2009:86-90 [FREE Full text] [Medline: 20351828]
Starren J, Johnson SB. An object-oriented taxonomy of medical data presentations. J Am Med Inform Assoc 2000;7(1):1-20[FREE Full text] [Medline: 10641959]
Hermanny K, Theis S, Christmann CA, Bleser G, Mertens AW, Dogangün A. Bewertung ausgewählter Studiendesigns zur
Untersuchung persuasiver Selbstmonitoringsysteme. In: Soziotechnische Gestaltung des digitalen Wandels - kreativ,
innovativ, sinnhaft : 63. Kongress der Gesellschaft für Arbeitswissenschaft : FHNW Brugg-Windisch, Schweiz, 15.-17.
Februar 2017 / Gesellschaft für Arbeitswissenschaft e.V. 2017 Presented at: 63. Kongress der Gesellschaft fürArbeitswissenschaft; Feb 15-17, 2017; Brugg-Windisch, Switzerland. [doi: 10.1007/978-3-319-58466-9_27]

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

Theis et al

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.70
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
53.

54.
55.
56.
57.
58.
59.

60.

61.

62.

63.

64.
65.

66.

67.
68.

69.

Theis et al

World Report on Ageing and Health. Geneva: World Health Organization; 2015. URL: http://apps.who.int/iris/bitstream/
handle/10665/186463/9789240694811_eng.pdf;jsessionid=9408D59CA7075996B40B1DE162215A0C?sequence=1[accessed 2018-05-22] [WebCite Cache ID 6zbNgYnye]
Sackmann R, Winkler O. Technology generations revisited: the internet generation. Gerontechnology 2013;11:493-503.
[doi: 10.4017/gt.2013.11.4.002.00]
Best SJ, Krueger BS. Internet Data Collection: Quantitative Applications in the Social Sciences. Thousand Oaks, CA:
SAGE Publications; 2006:104.
SurveyMonkey. URL: https://www.surveymonkey.de/ [accessed 2017-08-09] [WebCite Cache ID 6saegglPf]
clickworker. URL: http://www.clickworker.com/ [accessed 2017-08-09] [WebCite Cache ID 6saegglPf]
MAXQDA. Media review URL: http://www.maxqda.com/ [accessed 2018-05-22] [WebCite Cache ID 6zbTtfcie]
Theis S, Rasche P, Mertens A, Schlick CM. An age-differentiated perspective on visualizations of personal health data. In:
Schlick CM, Duckwitz S, Flemisch F, Frenz M, Kuz S, Mertens A, et al, editors. Advances in Ergonomic Design of Systems,
Products and Processes. Berlin: Springer Berlin Heidelberg; 2017:289-308.
Theis S, Bröhl C, Wille M, Rasche P, Mertens A, Beauxis-Aussalet E, et al. Ergonomic considerations for the design andthe evaluation of uncertain data visualizations. In: Human Interface and the Management of Information: Information,
Design and Interaction. Vol. Cham, Switzerland: Springer International Publishing; 2016:191-202.
Theis S, Mertens A, Schlick M. Preliminary results of an exploratory study towards a general task and data model fortelemedical visualizations. 2015 Presented at: 5th Triennial Congress of the International Ergonomics Association. Vol. 9;
Aug 9-14, 2015; Melbourne, Australia.
Browne SH, Behzadi Y, Littlewort G. Let visuals tell the story: medication adherence in patients with type II diabetes
captured by a novel ingestion sensor platform. JMIR Mhealth Uhealth 2015 Dec 31;3(4):e108 [FREE Full text] [doi:10.2196/mhealth.4292] [Medline: 26721413]
Berry N, Bucci S, Lobban F. Use of the Internet and mobile phones for self-management of severe mental health problems:
qualitative study of staff views. JMIR Ment Health 2017 Nov 01;4(4):e52 [FREE Full text] [doi: 10.2196/mental.8311][Medline: 29092809]
Fraccaro P, Arguello CM, Ainsworth J, Buchan I. Adoption of clinical decision support in multimorbidity: a systematic
review. JMIR Med Inform 2015 Jan 07;3(1):e4 [FREE Full text] [doi: 10.2196/medinform.3503] [Medline: 25785897]
Hansel B, Giral P, Gambotti L, Lafourcade A, Peres G, Filipecki C, et al. A fully automated web-based program improves
lifestyle habits and HbA1c in patients with type 2 diabetes and abdominal obesity: randomized trial of patient e-coaching
nutritional support (the ANODE study). J Med Internet Res 2017 Nov 08;19(11):e360 [FREE Full text] [doi:10.2196/jmir.7947] [Medline: 29117929]
Park DY, Goering EM, Head KJ, Bartlett Ellis RJ. Implications for training on smartphone medication reminder app use
by adults with chronic conditions: pilot study applying the technology acceptance model. JMIR Formativ Res 2017 Nov
10;1(1):e5. [doi: 10.2196/formative.8027]
Theis S, Bröhl C, Rasche P, Wille M, Schlick C, Mertens A. Age-dependent health data visualizations: a research agenda.
Mensch und Computer Workshopband 2016. [doi: 10.18420/muc2016-ws11-0003]
Chen AT, Carriere RM, Kaplan SJ. The user knows what to call it: incorporating patient voice through user-contributed
tags on a participatory platform about health management. J Med Internet Res 2017 Sep 07;19(9):e292 [FREE Full text][doi: 10.2196/jmir.7673] [Medline: 28882809]
Almalki M, Gray K, Martin-Sanchez F. Development and validation of a taxonomy for characterizing measurements inhealth self-quantification. J Med Internet Res 2017 Nov 03;19(11):e378 [FREE Full text] [doi: 10.2196/jmir.6903] [Medline:
29101092]

Abbreviations
ICT: information and communication technology

Edited by G Eysenbach; submitted 12.11.17; peer-reviewed by U Backonja, V West; comments to author 09.12.17; revised version
received 14.03.18; accepted 03.04.18; published 09.07.18.
Please cite as:
Theis S, Rasche PWV, Bröhl C, Wille M, Mertens A
Task-Data Taxonomy for Health Data Visualizations: Web-Based Survey With Experts and Older Adults
JMIR Med Inform 2018;6(3):e39
URL: http://medinform.jmir.org/2018/3/e39/
doi:10.2196/medinform.9394
PMID:29986844

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.71
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Theis et al©Sabine Theis, Peter Wilhelm Victor Rasche, Christina Bröhl, Matthias Wille, Alexander Mertens. Originally published in JMIR
Medical Informatics (http://medinform.jmir.org), 09.07.2018. This is an open-access article distributed under the terms of theCreative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The
complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright andlicense information must be included.

http://medinform.jmir.org/2018/3/e39/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e39 | p.72
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

Original Paper

Implementing a National Electronic Referral Program: Qualitative
Study
Marcella McGovern1, PhD; Maria Quinlan1, PhD; Gerardine Doyle1,2, PhD; Gemma Moore1, PhD; Susi Geiger1,2,
PhD
1

Applied Research for Connected Health, University College Dublin, Dublin, Ireland

2

UCD College of Business, University College Dublin, Dublin, Ireland

Corresponding Author:
Marcella McGovern, PhD
Applied Research for Connected Health
University College Dublin
G41, NEXUS UCD
Dublin, D04 V2N9
IrelandPhone: 353 1 716 5407
Email: marcella.mcgovern@ucd.ie

Abstract
Background: Electronic referrals or e-referrals can be defined as the electronic transmission of patient data and clinical requests
between health service providers. National electronic referral systems have proved challenging to implement due to problems offit between the technical systems proposed and the existing sociotechnical systems. In seeming contradiction to a sociotechnicalapproach, the Irish Health Service Executive initiated an incremental implementation of a National Electronic Referral Programme(NERP), with step 1 including only the technical capability for general practitioners to submit electronic referral requests to
hospital outpatient departments. The technology component of the program was specified, but any changes required to embedthat technology in the existing sociotechnical system were not specified.
Objective: This study aimed to theoretically frame the lessons learned from the NERP step 1 on the design and implementationof a national health information technology program.
Methods: A case study design was employed, using qualitative interviews with key stakeholders of the NERP step 1 (N=41).
A theory-driven thematic analysis of the interview data was conducted, using Barker et al’s Framework for Going to Full Scale.
Results: The NERP step 1 was broadly welcomed by key stakeholders as the first step in the implementation of electronic
referrals—delivering improvements in the speed, completeness of demographic information, and legibility and traceability ofreferral requests. National leadership and digitalized health records in general practice were critical enabling factors. Inhibiting
factors included policy uncertainty about the future organizational structures within which electronic referrals would be implemented;
the need to establish a central referral office consistent with these organizational structures; outstanding interoperability issues
between the electronic referral solution and hospital patient administration systems; and an anticipated need to develop specialist
referral templates for some specialties. A lack of specification of the sociotechnical elements of the NERP step 1 inhibited thenecessary testing and refinement of the change package used to implement the program.
Conclusions: The key strengths of the NERP step 1 are patient safety benefits. The NERP was progressed beyond the pilot
stage despite limited resources and outstanding interoperability issues. In addition, a new electronic health unit in Ireland (eHealth
Ireland) gained credibility in delivering national health information technology programs. Limitations of the program are its poor
integration in the wider policy and quality improvement agenda of the Health Service Executive. The lack of specification of thesociotechnical elements of the program created challenges in communicating the program scope to key stakeholders and restricted
the ability of program managers and implementers to test and refine the change package. This study concludes that while thesociotechnical elements of a national health information technology program do not need to be specified in tandem with technicalelements, they do need to be specified early in the implementation process so that the change package used to implement the
program can be tested and refined.
(JMIR Med Inform 2018;6(3):e10488) doi:10.2196/10488

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.73
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

KEYWORDS
electronic referrals; scale-up; eHealth; implementation; policy

Introduction
Electronic Referrals
Electronic referrals or e-referrals can be defined as the electronictransmission of patient data and clinical requests between healthservice providers [1]. Shifting from paper-based referrals (ie,postal letter or fax) to electronic referrals offers the opportunity
to transform the interface between primary and specialty care[2]. Historically, the default clinical request from a generalpractitioner (GP) referring to a specialist was to request aface-to-face consultation with a given service user [3]. Electronic
referral technology, however, can support a two-way channelof communication between referrer and referee, creating theopportunity for more flexible and consultative forms of dataexchange and clinical requests [4]. In addition, electronic
referrals provide health systems with the capability to optimize
system capacity, whereby GPs can be supported by specialists
to care for service users in the community until they genuinely
require a specialist appointment [5].

Development of Electronic Referrals in the Irish HealthService Executive
The initiation of an Irish electronic referral pilot program in
January 2011 was not solely motivated by the potential forelectronic referrals to transform the interface between primaryand specialty care. A crisis emerged in March 2010, when the
media reported that one of Ireland’s largest hospitals had 30,000
unopened or unprocessed GP outpatient referrals. The Irish
Health Service Executive (HSE) commissioned an investigation[6] and the Health Information and Quality Authority (HIQA)
partnered with the Irish College of General Practitioners (ICGP)
to conduct a review of the referral management between GPs
and hospital outpatient departments (OPDs; GP-OPD). This
HIQA-ICGP partnership created a standardized general referral
template, specifying the essential information that needs to becontained in a referral from a GP to a hospital OPD. In addition,
the HIQA-ICGP partnership recommended that theirstandardized template could form the informational basis foran electronic referral solution between GPs and OPDs [7].
Meanwhile, an advisory group had been established in the Southof the country, comprising clinical, management, information
technology (IT), and patient representatives, to reconfigure
hospital services in that region. The group partnered with
Healthlink—an Irish structured health care messaging
platform—to develop and pilot an electronic referral pathway
between GPs and OPDs for 7 hospitals in their region. The pilot
project revealed several challenges for implementing an
end-to-end electronic referral solution, capable of offering a
two-way interface between GPs and hospital OPDs. Foremostof these challenges were the outstanding interoperability issues
between the Healthlink platform and hospital patient
administrative systems (PAS); second, the human resourcing
of hospital central referral offices (CROs) to process electronic
referrals.

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

Despite these obstacles to the implementation of an end-to-end
electronic referral solution, the pilot project successfully
established the technical capability, through the Healthlink
platform, for GPs to electronically submit their referrals to
hospital OPDs. This first step in the electronic referral process
has been described as the electronic referral request [8].
Furthermore, the pilot project found that the use of electronic
referral requests resulted in the following improvements: (1)improved legibility because all information is typed in astandardized template, (2) improved completeness of data
because of the mandatory fields in the standardized template,(3) assurance for GPs and patients that their referral had been
received because an automated email is returned to the referring
GP once it is digitally opened in the hospital, and (4) improved
traceability and visibility for hospitals in the referralmanagement because Healthlink creates a digital record of whenand how many electronic referrals have been received by eachhospital, and when they were triaged [9].
These simple and yet important patient safety benefits informed
a decision by a newly established unit in the HSE, called eHealth
Ireland, to establish a National Electronic Referral Programme(NERP) with step 1 involving the scale-up of the technicalcapability for GPs to submit electronic referral requests to
hospital OPDs.

Impact of Scale on a Sociotechnical Approach
Reviews from some of the earliest deployments of national
electronic referral systems, including Norway [10], theNetherlands, and Denmark [11], recommend a sociotechnicalapproach to implementing electronic referrals. A sociotechnicalapproach considers how the technical features of a healthinformation system interact with the social features of a healthcare work environment [12]. eHealth Ireland ’s decision toinitiate the NERP on an incremental basis, that is, step 1 with
only the technical capability for electronic referrals, appears to
be at odds with this sociotechnical approach. The only target
specified was to establish, within 12 months, the technicalcapability for GPs to submit electronic referral requests to atleast one OPD specialty in all hospitals. Regarding the adoptionof that technical capability, no targets were set for which OPD
specialties would be included in the program, what the target
volume of electronic referrals versus paper referrals would be,or what proportion of GPs would be engaged. That is, thetechnology component of the program was specified, but theextent at which that technology would interact with thesociotechnical system was not specified.
In the UK, Eason applied a sociotechnical lens to theimplementation of the National Health Service’s National
Programme for IT and highlighted that in national health IT
programs, the desired technical and social systems are notdesigned and implemented simultaneously [13], as suggested
by many interpretations of a sociotechnical theory approach[14].

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.74
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

Figure 1. Theoretical framework—Barker et al’s [16] Framework for Going to Full Scale (reprinted with permission from P Barker).

Instead, standard technical systems are predefined at the national
level, and flexibility needs to be provided for local
implementation sites to adopt technical systems in ways thatmeet local needs and enable them to engage in sociotechnicalsystems design at a level where the local user community canplay a full part [13]. This suggestion that the sociotechnicalapproach needs to be modified for national health IT programs
is reflected in the design of the NERP step 1 [15], where thetechnical system was predefined nationally, but it was left tolocal implementation sites to undertake the sociotechnicalsystems design work. However, it remains unclear from Eason’s
critique of sociotechnical systems theory if or when thesociotechnical elements of a national health IT program should
be defined at the national level [13].

Study Goal
To contribute to this discussion on the design andimplementation of national health IT programs, this paper
presents the findings from qualitative, in-depth interviews
conducted with key stakeholders in the implementation of the
NERP step 1. The following two research questions seek toexplore the arguments for and against progressing with thescale-up of only the technical elements of a national health IT
program, using Ireland’s NERP step 1 as an empirical casestudy: (1) What were the strengths and limitations of thescale-up of the NERP step 1, as a technical-only intervention?
and (2) Do the sociotechnical elements of a national health IT
program need to be specified at the national level?
This study aims to theoretically frame the lessons learned from
Ireland’s NERP step 1 for policy makers and implementersseeking guidance on how to design and implement national
health IT programs.

Theoretical Framework
We adopted the Institute for Healthcare Improvement’s
Framework for Going to Full Scale [16] as the theoreticalframework to guide our empirical inquiry and analysis. This
framework proposes that to take a health care quality
improvement to full scale, it is first necessary to account for thefactors required to promote the adoption of changes and support
scale-up, and second, to design at the outset a phased plan toreach full-scale implementation.
The phases of scale-up proposed by this framework for healthcare quality improvement include: (1) Set-Up; (2) Develop the
Scalable Unit; (3) Test Scale-Up; and (4) Go to Full Scale. Each
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderXof these phases is either enabled or inhibited by the availabilityof certain adoption mechanisms and support systems (Figure1). Adoption mechanisms to account for include better ideas,
leadership, communication, policy, and a culture of urgency
and persistence. Support systems include human capability forscale-up, infrastructure for scale-up, data collection andreporting systems, a learning system, and the need to design forsustainability.
An important reason for selecting this theoretical framework isthat it can accommodate the NERP’s incremental design,whereby this study only examines step 1 in the implementationof electronic referrals and not the complete implementation ofelectronic referrals. The phases of scale-up in this framework
are informed by Plan-Do-Study-Act (PDSA) cycles of quality
improvement. It is not assumed that what is being implemented
is the complete program. A PDSA cycle requires only that for
any given program or program component, a theory of change
can be specified and then tested across a range of contexts before
being implemented at full scale. The framework contains a
feedback loop so that the first three phases of scale-up can berevisited and adapted if new learnings at a later phase revealthat an adaptation would optimize the scale-up. Figure 1 presents
this feedback loop by the counter-clockwise arrows above thefour phases of scale-up.

Methods
Methodological ApproachThis study explored the implementation of the NERP step 1
using qualitative, in-depth interviews with key program
stakeholders. This approach captures individual participants’experiences, narratives, ideas, and discourses [17] and informs
an analysis of the scale-up strategy employed.

Recruitment
Ethics
Ethical approval was granted by the Office of Research Ethics
in University College Dublin (UCD). No vulnerable populations
participated in this study, and no patient data were collected.
All participants were interviewed in a professional capacity as
stakeholders in the scale-up of electronic referrals in Ireland.
Participant anonymity and confidential data management were
the dominant ethical considerations for this study and weremaintained in line with UCD Research Ethics Guidelines.
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.75
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

ParticipantsThis study investigated the lessons to be learned from the NERP
step 1 on scaling-up only the technical elements of a national
health IT program. Access to 1 of 7 pilot sites and 5 of 42 sites
targeted by the NERP step 1 was facilitated by eHealth Ireland.
Although not randomly selected, the 5 NERP step 1 sites
included public and voluntary hospitals, as well as regional andurban hospitals, providing a broad representation of
implementation sites. The key inclusion criterion for recruiting
participants was as follows: Has this stakeholder been involvedin the design or implementation of the NERP step 1? If not, it
was considered unsuitable for stakeholders to serve as key
informants on the strengths or limitations of a technical-onlyscale-up or in offering an empirical assessment of whether thesociotechnical elements of a national health IT program shouldbe specified at the national level. Based on this criterion, we
did not include service users and hospital specialists in our study
design because they were not directly involved in designing or
implementing the program at this early step 1 stage of the NERP.
However, studies of later stages of the implementation should
include these crucial stakeholders, where, for example, service
users might have access to a Web-based appointment portal, or
hospital specialists might be engaged to design specialist referral
templates and therefore, will be in a position to speak about
their experiences of designing and implementing the program.
The following participants were recruited from the pilot site:
pilot management (n=3); hospital administration or management(n=3); general practice (n=3); and information communication
technology (ICT; n=3). In addition, the pilot general practiceand ICT stakeholders were involved in the NERP step 1 andtherefore appear in Table 1 as both pilot and NERP
national-level stakeholders. Moreover, the NERP national-level
stakeholders included NERP management (n=2) and other HSE
stakeholders (n=3) who were involved in the design andimplementation of the NERP step 1. At the implementation site
level, we recruited 4 additional general practice stakeholders,

including 3 GPs and 1 general practice secretary, bringing thenumber of GPs who participated in the study to 6 out ofapproximately 3000 GPs [18] operating in the Irish healthsystem. Furthermore, participants recruited from within the
NERP step 1 hospital sites included the following: hospitaladministration or management (n=17) and hospital ICT (n=3).
As a qualitative study, participants (N=41) were recruited as
key stakeholders and informants on the design andimplementation of the NERP step 1; they did not constitute a
representative sample of their peers in the Irish health system.
Overall, 28 interviews were scheduled. Of 41 participants, 19participated in face-to-face group interviews and the remaining
22 were interviewed individually. The group interviews involved
a range of 2-5 participants and were predominantly undertakenwith the hospital administration or management stakeholder
group. Of 22 interviews conducted with individual participants,
5 were conducted via telephone and 17 on a face-to-face basis.
All participants consented to have their interview recorded at
the outset, using a digital voice recorder.

Data Analysis
We conducted a thematic analysis, using Braun and Clarke’s[19] 6-phase procedure for thematic analysis and Barker et al’s
[16] framework to organize the data from a scale-up perspective.
Thematic analysis is a method for identifying, analyzing, andreporting patterns (themes) within data [19]. The audio
recordings of all 28 interviews were transcribed by theinterviewing author (GM). Thematic analysis does not requirea full verbatim transcription, including nonverbal cues (eg,silences, body language, and external noises) or emotional
aspects (eg, laughs, coughs, and sighs) [20]. Such data would
have contributed little toward answering this study’s researchquestions but would have taken a substantial amount of timeand resources. All transcripts were cross-checked by the leadauthor (MMG) who audited transcription by listening throughall audio recordings while reading the transcripts.

Table 1. Participant involvement in the National Electronic Referral Programme (NERP; N=41).
Key stakeholder types

Involvement, n (%)
Pilot (n=12)

NERP (n=35)

Both (n=6)

Pilot management

3 (25)——

Hospital administration or management

3 (25)——NERP management

—

2 (6)

—

Health Service Executive

—

3 (8)

—

Information communication technology

3 (25)3 (8)

3 (50)

General practice

3 (25)3 (8)

3 (50)

Hospital administration or management—

17 (49)—

Information communication technology

—

3 (8)

—

General practice—

4 (11)—Pilot

NERP national level

NERP implementation sites

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.76
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Furthermore, any points of divergence in the interpretation of
how the spoken word should be written in the transcripts were
documented and later discussed by MMG and GM to obtain anagreement on transcription.
In the first analytical step, we conducted an inductive thematic
analysis [19]. Overall, 149 initial codes were generated andcollated into 7 initial themes as follows: (1) Stakeholder
Consultation (12 codes); (2) Scope and Pace of Change (15codes); (3) Technological Design (31 codes); (4) Organizational
Change (25 codes); (5) Engagement (18 codes); (6) Quality
Improvement (37 codes); and (7) Irish Context (11 codes). The
two research questions aimed to explore the arguments for andagainst progressing with the scale-up of only the technicalelements of a national health IT program. Among the diverse
stakeholders interviewed, contradictory perspectives werearticulated on whether or not it was the correct decision toprogress to the large-scale implementation with a technical-only
solution.
In the second analytical step, Barker et al’s Framework for
Going to Full Scale was identified as a framework suitable for
guiding a more theoretically driven thematic analysis [19] ofthe data, which could then be overlaid with the inductive coding.
Coding and analyzing the data within this framework’s phasesof scale-up—each of which was either enabled or inhibited by
specific adoption mechanisms and support systems—produceda coherent and accurate representation of the diverse
perspectives expressed in the data. An initial, comprehensive
report was produced by MMG, using the data coded under eachtheme to present a theory-driven response to the two researchquestions. Through an iterative process of review and revisions
involving first, MQ and GM, and then GD and SG, the authors
reached consensus on the final structure of the findings and theselection of representative quotes. This collaborative processfacilitated an investigation of any contradictory evidence or
possible alternative interpretations of the data to ensure theminimization of individual bias in the results presented.

Results
Data Presentation
This section presents data collected via qualitative in-depth
interviews with key stakeholders of the NERP step 1 using thetheoretical structure provided by Barker et al’s Framework for
Going to Full Scale. Figure 2 illustrates a stakeholder map of
the number and type of stakeholders, who informed these results
and where they come in the process flow of the NERP step 1.
Figure 3 illustrates the application of Framework for Going toFull Scale to this study’s data, by indicating the phases ofscale-up that each adoption mechanism and support systememerged in the analysis, as an enabler or inhibitor of scale-up.
Finally, to provide some context for these results, Multimedia
Appendix 1 charts the number of electronic referrals submitted
to the pilot hospital OPD and the 5 NERP hospital OPDs who
participated in this study in 2015 and 2016, when NERP step
1 was implemented.

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

McGovern et al

New Scale-Up Idea
The phases of scale-up are triggered by the discovery of a newscale-up idea or a new best practice, which is perceived by the
stakeholders as a “better idea” (adoption mechanism). A
national-level participant commented that the NERP step 1 was:...an easy sell, [its] patient safety...a solution has beendeveloped so it’s a matter of taking the solution androlling it out to different acute hospitals. [Participant
28]
Similar to the pilot experience, the NERP step 1 participants
cited speed, complete demographic information, legibility, andtraceability as the 4 key patient safety improvements deliveredin the NERP step 1. Participants commented that “speed ofreferral would be the biggest thing...The GP knows it’s got
here” (Participant 10) because an automated notification isreturned to the GP once the electronic referral has been opened
within the hospital.
It’s instant, it sells itself. You send in the referral, thehospital has it, there’s no post, you’re not waiting a
day for it to be delivered. [Participant 29]
It’s good they have a minimum data set...Like we’ll
never be missing a date of birth or [receive]...only
one line of an address...They will always give you aphone number on it. [Participant 40]
Another hospital administrator pointed out that full contact
details, including mobile phone numbers, are very important sothat they can “text remind people...to help reduce the DNA (Do
Not Attend) rate” (Participant 5). Finally, several participantscommented on the legibility benefits of electronic referrals inthat they save time trying to decipher difficult hand-writing or
calling GP surgeries to confirm details or to seek missing
information.
It’s legible you know. Many times you have to ringthem up [GP surgeries]. [Participant 40]
However, with electronic referrals, they “can find a patientmuch easier on the system now” (Participant 24).
One important sociotechnical element built into the pilot
program had been the requirement for hospitals to return a triage
outcome message, via Healthlink, to referring GPs. Manpowerplanning issues identified in hospital CROs during the pilot
stage resulted in the GP triage outcome message not being
specified as an element of the NERP step 1. One of the pilot
GPs commented that:...it remains to be seen...how negative that will be...It
reduces the communication back to the GP, and it
doesn’t tell the GP how the patient has been triaged.
[Participant 16]
A national-level participant commented that:...we thought the GPs would be up in arms and theywould go crazy about it but actually when we did goback to the ICGP...they said, “we’d be disappointed
but at the same time...we would prefer that they[hospitals] went with it without responses than not
go at all.” [Participant 12]
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.77
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

Figure 2. The stakeholder map. GP: general practitioner, ICT: information communication technology, NERP: National Electronic Referral Programme.

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.78
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

Figure 3. Application of the Framework for Going to Full Scale to the NERP step 1.

As such, GPs supported the NERP step 1 in proceeding withthe scale-up of the technical capability for all hospitals to receive
electronic referral requests, with the knowledge that hospitals
did not have the “human capability for scale-up” (support
system) to support an electronic processing of referrals (ie,eTriage and eAppointments). This acceptance among key
stakeholders that the NERP step 1 is only about paving the way
for the complete implementation of electronic referrals is also
captured in a comment by a national-level participant who
described the NERP step 1 as getting: “...the footprint ofeReferrals out to all hospitals around the country” (Participant
18).

Set-Up
Set-Up is the first phase of scale-up, where the ambition for
full scale is defined. Limiting the scope of the NERP step 1 tothe technical capability was explained by national-levelparticipants as a pragmatic decision. A national-level stakeholder
explained that they did not “have the bandwidth within [their]
resources to go to each site” to support a sociotechnicalimplementation.
When we are asked to rollout eReferrals within 12
months, what we can do is we can put the capability
in place for each of the sites. [Participant 18]
In terms of the “ask” to rollout electronic referrals in 12 months,
the “leadership” provided by the HSE’s new eHealth Ireland
unit was widely cited as a critical adoption mechanism for the
set-up of the NERP. One pilot participant suggested that the“timing was impeccable” (Participant 28) for the appointmentof a chief information officer (CIO) to lead the eHealth Irelandunit.
If he hadn’t arrived I would say that at this stage,
we’d have rolled it out in the South or Southwest
Hospital Group [pilot] and possibly no further.
[Participant 28]
A national-level participant commented that:...this is the first...of any of the projects we’ve done,
where there’s been a national focus. Where from the
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

top, it’s been said, “everyone has to accept electronic
referrals by X date.” [Participant 12]
In addition, a GP who was involved in the pilot commented thatthe new CIO was “providing a vision for where the service
needs to go” but...
...there’s a huge amount of work that needs to be done
and huge investment that needs to take place, and I
suppose that remains to be seen, whether that will beavailable. [Participant 16]
This last comment suggested that leadership requires not only
vision but also the ability to secure funding. A comment by anational-level stakeholder supported this suggestion in referringto the “credibility piece,” whereby:...if we can deliver a project of this type in a timelinethat’s considered sensible...then it provides more
confidence that the Office of the CIO and the
Healthlink team can actually deliver significant
change in a reasonable amount of time. [Participant
18]
This comment provided context for the ambitious 12-monthtime-frame to scale-up the technical capability for electronic
referrals, particularly because further resources need to be
secured to proceed beyond step 1 of the NERP.
Second, defining the ambition of the NERP step 1 as putting
the technical “capability” in place for all hospitals to receive
electronic referral requests from GPs indicated confidence thatGPs will submit their referrals electronically, should this facility
be available to them. The level of digitalization of generalpractice health records represents an important “infrastructure
for scale-up” support system. The Healthlink solution is fully
integrated into general practice software packages, and GP
participants emphasized the importance of this integration in
their interviews.
[GPs] have all the information in the system andbeing able to extract it and package it up and send it
off electronically is kind of a side effect of theinvestment that they’ve [GPs] made over the years.
[GP 1]JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.79
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
No formal incentives are offered to GPs to use electronic referral
requests and, therefore, their adoption of the solution relies upon
their technical capability to submit electronic referrals and theperception that this solution is a “better idea.”

Develop the Scalable Unit
The second phase of scale-up is developing the scalable unit,
which is the smallest unit of the system targeted for the full-scale
implementation. This analysis proposes that the scalable unitof the NERP step 1 should specify the proportion of (1) hospitals
targeted, (2) OPD specialties targeted, (3) electronic versus
paper referral requests targeted, and (4) GPs targeted to use
electronic referral requests. In practice, the scalable unit
specified by eHealth Ireland only included the first of theseelements, with a full-scale target of all public hospitals.
Implementing this first element was the focus of the NERP step
1 throughout this study period (October 2015-May 2016) and
was achieved in May 2016, after 17 months of implementation,
when (1) all public hospitals had (2) at least one specialty
accepting (3) outpatient electronic referrals from (4) referring
GPs. With all hospitals targeted, a minimal specification of atleast one OPD specialty in each hospital can be assumed.
However, no specification was provided for the target proportionof electronic versus paper referral requests or the proportion of
GPs targeted to use electronic referral requests. The remainderof the description of the NERP step 1’s progression through thephases of scale-up will, therefore, be dealing with what remains
to be scaled rather than what has been scaled.
Interview data collected suggested that the incomplete
development of the scalable unit reflects uncertainty in national“Policy” (adoption mechanism). The Irish health system isundergoing a process of de-centralization, from a highly
centralized HSE to the creation of hospital groups andcommunity healthcare organizations [21,22]. These structures
are yet to be finalized [23], creating a challenge for the NERP
step 1 because it is envisioned that CROs will ultimately becreated at the hospital group level, rather than within individual
hospitals [24]. Arguably, it would be a duplication of effort toimplement the sociotechnical process changes associated withelectronic referrals at the hospital level only for those processes
to be changed again once there is certainty about the hospitalgroups. A national-level stakeholder highlighted that it is not adecision for eHealth Ireland whether outpatient electronic
referrals will be managed in each hospital or at a hospital grouplevel.
[It is] not something that IT can make a call on...we’ll
certainly drive it once we’re clear this is a directionthat is best for the patient and for the service.
[Participant 18]
This quote illustrated a governance challenge faced by national
health IT programs like NERP, in that the authority to makekey decisions about the design of such programs might lie
outside of the program team.
In addition, shortcomings emerged at this phase in the“communication” adoption mechanism. Barker et al suggested
that it is necessary to communicate the value of a scale-up toboth leadership and implementers, ideally by providing real-time
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

McGovern et aldata from one scale-up phase to garner support for the next
phase [16]. Among implementers, hospital administrative staff,
in particular, reported a lack of engagement or opportunity tocontribute to developing the scalable unit, which negatively
affected their “sense of ownership of it” (Participant 37). In
addition, CRO staff commented that:...there are meetings and different groups but...theadministrative end is not heard all the time...theycome looking for secretarial support but then nobudget or nothing available. [Participant 23]
Perhaps, stemming from this lack of consultation, there was anunmet expectation from CRO staff that electronic referrals weregoing to save them time.
Everything that’s been done electronically, it’ssupposed to save time and resources but actually it
doesn’t. It does exactly the opposite mostly.
[Participant 22]
While electronic referrals will reduce the administrative burden
for CRO staff once the interoperability issues between
Healthlink and PAS are resolved, in the short term, it increases
the workload because they have to manage an additional modeof referral request, in addition to traditional post and fax.
Furthermore, participants reported that the strategy forcommunicating the value of electronic referrals to GPs requires
clarification to increase the proportion of GPs submitting
electronic referral requests. This element of the scalable unit
was not specified for the NERP step 1, and therefore, nobody
was officially tasked with the responsibility to increase theproportion of GPs using electronic referrals. A hospital-based
implementer suggested that:
I felt like why was I having to try to promote
Healthlink?...Nobody could give me any
communication tools to use for the GPs—so we had
to try and figure out the best way to do it. [Participant29]
National-level participants suggested that ultimately, GPs are“independent sole traders” (Participant 27). The key authority
capable of shaping the referral behaviors of GPs are the hospitals
receiving those referrals.
That is really, a HSE hospital led kind of
initiative...[The] ultimate step would be formanagement to say, “this is how we want you torefer”...unless there is an exception. [Participant 27]
In this study, participant GPs broadly agreed with thisperspective, suggesting that local hospitals are in the best
position to change GPs’ referral behaviors together with local
peer promotion through the ICGP’s Continuing Medical
Education meetings. GPs commented that:
I think it would be great to see the hospitals runningwith the ball on this one alright. [Participant 13]
They also said, “...people listen to their peers” (Participant 39).
These comments illustrated the importance of acknowledging
GPs’ professional independence in the Irish health system, when
designing a communication strategy to engage GPs in thescale-up of a national IT program.
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.80
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Test Scale-Up
Test Scale-Up is where the underlying theory of change and thechange package are tested in a broader range of settings to refine
program hypotheses and build the belief and will of leaders andfrontline staff to support the changes [16]. To progress theimplementation beyond the NERP step 1, developing appropriate
data collection and reporting systems (Support System) would
require a more complete development of the scalable unit.
Participants reported that electronic referrals present an
opportunity to standardize what data are collected (Participant7), especially through automatically populating demographic
information (Participant 40], clinical history (Participant 22),and medication and allergy information [(Participant 29) fromthe GPs’ medical records. In addition, participants reported thatelectronic referrals are:...giving greater visibility on referral volume, referral
tracking, all those sorts of things by specialty within
hospitals. [Participant 27]...whereas before we were relying on staff members
putting them in an Excel...So now every referral tobe processed must be on PAS. [Participant 5]
The disadvantage of structured messaging is that it might limit
GPs’ ability to communicate details about a referral. One GP
commented that:
You can write a very good clinical note using freetext, probably the best quality clinical notes because
it captures what the patient and yourself are saying.
You can’t do that with something that’s completely
structured. When you’re picking from drop-downmenus or whatever. [Participant 30]
Moreover, participants highlighted the importance of buy-infrom stakeholders on the type of data collected. One participant
commented that:
I maintain that no clinician wants to work to apolitical target...They don’t mean anything clinically[Participant 35]
A pilot participant cautioned that:...when you use data in a punitive way...people areresistant to it. [Participant 19]
These quotes highlighted the potential for electronic referrals
to greatly improve the volume and quality of data collected onreferral management as well as the importance of engaging withstakeholders to determine what data would offer the mostconstructive and meaningful insights for the quality
improvement.
Regarding reporting systems, participants reported receiving amonthly Healthlink escalation report, showing electronic referral
requests that were received by the hospital but for which notriage outcome had been logged on Healthlink. Although logging
the triage outcome to Healthlink was beyond the scope of the
NERP step 1, participants reported that this report supports
CRO staff in tracking and tracing electronic referrals.

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

McGovern et al
It has actually highlighted that we weren’t doing it[managing referrals] as well as we thought we were
doing it. [Participant 22]
Participants described two other national programs to which
they submit data and receive reports relevant to electronic
referrals, namely the HSE’s Outpatient Services Performance
Improvement Programme (OSPIP) and an independent statutory
body called the National Treatment Purchase Fund (NTPF).
Crucially, however, neither OSPIP nor NTPF targets areformally aligned with any specific targets for the NERP step 1.
A CRO participant commented that “in the Health Service,
there’s no picture of what’s happening” (Participant 22).
This suggests a lack of data feedback to implementers, either
on the NERP step 1 on its own, or a more strategic data reporting
system that utilizes the data collected across HSE and statutory
programs.
This lack of development in data collection and reporting
systems exerts knock-on effects on the “learning system”(support system) for the NERP step 1. “Large-scale change
requires a mechanism for collecting, vetting, and rapidly sharing
change ideas or interventions”...to assemble a “change package”
for scale-up [16]. Participants interviewed indicated that
informal learning from the pilot sites was encouraged by thenational implementation team, but no evidence emerged of any
formal learning system. A national-level stakeholder commented
that as part of the “go-live” training, it is:...normally suggest[ed] that they [implementers]
speak to other counterparts in other hospitals thathave already gone live. [Participant 18]
Furthermore, one implementation site participant commented
that he had “two conference calls” (Participant 21) with amember of the pilot implementation team to learn from theirexperience, but otherwise, the data did not suggest that alearning system was in place for the NERP step 1.
Besides these support systems, an important adoption
mechanism at this phase is a “culture of urgency andpersistence” (adoption mechanism), motivating stakeholders totake action and sustain their efforts to take the program to fullscale. A troubling theme emerged around how legacy IT failures
have created a culture of caution rather than urgency for national
health IT programs. One pilot participant explained that “there’s
the legacy belief around HSE ICT projects fail” (Participant38), and a national-level participant referred to how:...some of them are not open to new stuff because
they’ve been burnt in the past...Most sites needreassurance as to the impact it’s [NERP step 1] likely
to have operationally for them. [Participant 18]
Conversely, CRO participants reported that while they werecautious about electronic referrals, now that they are using
Healthlink, they find it very straightforward to use, and there
is a strong appetite for the implementation of an end-to-end
electronic referral solution. In addition, one participant
commented that:

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.81
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Now that we know how easy it is to go electronic, itwould be amazing to cut out all the filling. [Participant
4]
Similarly, another CRO participant commented that:...rather than sitting on this for a year and everyone
would just get too complacent with it and then it’smore change...If you’re in the middle of a project andthere’s more coming on board, you just take it.
[Participant 25]
These comments highlighted the importance of developing acomplete, scalable unit, whereby participants are clear on what
the vision for full scale is and they can then maintain momentum
in going to full scale.

Go to Full Scale
Go to Full Scale is the fourth and final phase of the Framework
for Going to Full Scale. This is the rapid deployment phase inwhich a well-tested set of interventions, supported by a reliabledata feedback system, is adopted by frontline staff on a larger
scale [16]. “Design for sustainability” is a critical adoption
mechanism for reaching this fourth phase of scale-up, whereby
throughout the 3 activity phases (Develop the Scalable Unit,
Test Scale-Up, and Go to Full Scale), the learnings aboutsustainability are used to refine the change package.

McGovern et al
been upgraded from the old PAS to an Integrated PatientManagement System (iPMS) can be integrated with Healthlink.
The HSE Integrated Patient Management System (IPMS) teamis working to “incorporate that functionality...but again, it's
just purely staff dependent” (Participant 32) as this team does
not have the human capacity to keep all hospitals technologicallyand procedurally up-to-date with the latest version. A CRO
participant claimed that:...it is a great system (new iPMS)...if the correct
processes were in place, it would be perfect.
[Participant 2]
The process changes involved in implementing iPMS require
CRO staff training. CRO staff reported that while the HSE IPMS
team did train on-site trainers, these “trainers only had a shortperiod of time to get trained themselves” (Participant 22), andas a consequence, the training “wasn’t specific to your job, it
was a general training group everyone went to” (Participant2). Moreover, upgrading hospital PAS to iPMS and providing
the necessary training to CRO staff on how to use this upgrade
are objectives beyond the scope of the NERP step 1. It is
important to highlight, however, that to progress beyond thestep 1 of the NERP (ie, eTriage, eConsult, eAppointments,
ePrescribing, and eDischarge), this interoperability issue must
be addressed.

Policy uncertainty is a key sustainability issue for the NERP
step 1, which has already been described above as theuncertainty about whether to proceed with the sociotechnicalprocess changes for electronic referrals at the hospital level [25]or to postpone process changes until the hospital group CROs
can be established [24]. An associated sustainability issue is thereconfiguration of administrative staff to work within newly
established CROs. Historically, each hospital consultant would
have their own secretary who manages referrals sent to that
consultant. A national-level participant suggested that theconsultant-level referral management creates “a lot of
duplication” because secretaries “wouldn’t be at full capacity
all the time” (Participant 28). Such a reconfiguration of staff isperceived as a challenge at the implementation level, whereparticipants explained that:

Furthermore, consultant engagement was beyond the scope of
the NERP step 1 because electronic referrals were printed once
they reached the hospital. Implementing a more complete
electronic referral solution will require hospital consultants
triaging electronic referral requests online. An implementationsite participant explained that they have had consultants fromcertain specialties requesting that “their own referral form” beaccommodated within Healthlink to enable the collection of
specialty-specific information to inform triage decisions. The“way we got around” that was by saying:...resources are still an issue with the Central Booking
and the Central Office. So, to do it from within yourcurrent compliment [of staff] initially is difficult.
[Participant 5]
Similarly, another CRO participant commented that:...we get a consistent message from the hospitals thatthey’d like to do more in the way of specialist referral.
[Participant 18]
The challenge is that consultants throughout the country “haveto agree with those extra parameters [for the electronic referral
template] that are unique for that speciality” and then “avoid
a scenario that says well actually ideally we’d like 20 extra
parameters from a GP” (Participant 18). An
implementation-level participant commented that:

We do [have a CRO] only we have no one to sit in it.
That’s why it comes to me. I’m the central office.
[Participant 40]
The variation in terminology used by participants to refer to the
CRO in the above quotes reflects the variation in set-up andfunctions of these offices across sites. This variation helps to
explain why some hospitals experience greater difficulty thanothers in implementing electronic referrals, if theiradministrative staff has not been reconfigured into a CRO.
The third key sustainability issue for the NERP is the persisting
interoperability issues between Healthlink and the hospital PAS.
One national-level participant explained that hospitals that have
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX...well Healthlink said they would take on certain
forms but if we could just run with this...and see how
we get on with it. [Participant 21]
Similarly, a national-level participant commented that:

I know that Healthlink did have some issues with[specialist referral forms]. They just want toconsolidate as much as possible. [Participant 21]
The Healthlink’s reluctance to integrate multiple specialist forms
may, however, be driven by technical and financial obstacles
to working with GP software vendors rather than a concern
about whether or not GPs would have the time to complete
specialist referral templates. Some national-level participantssuggested that:
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.82
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSEngaging with the vendors is a challenge...because
we’re very reliant on them doing the initial work to
get their products modified. [Participant 12]
Healthlink worked with the ICGP’s national General PracticeInformation Technology (GPIT) group “to do a specification
for the vendors...and coming up with agreements on cost andimplementation time-frames” (Participant 12). For Healthlink
to integrate specialist referral templates, it would again need
the support and cooperation of the ICGP GPIT group to engage
with GP vendors to make the necessary upgrades. If specialist
referral templates are perceived by GPs as unnecessarily
burdensome, the ICGP GPIT group might not be willing or able
to support this integration work in future.

Discussion
Principal Findings
This study aimed to theoretically frame the lessons learned fromthe NERP step 1 on the design and implementation of a national
health IT program. The NERP step 1 presented an interestingcase study of implementing a national health IT program because
it explicitly committed to a technical-first implementation rather
than a sociotechnical approach. A key strength of the program
was that it was welcomed by most key stakeholders as the firststep in the implementation of electronic referrals, delivering
important patient safety benefits. A national implementation ofelectronic referrals was progressed, despite limited resourcesand outstanding interoperability issues. In addition, it gained
credibility for a new eHealth Ireland unit, which demonstrated
that it can deliver national health IT programs. Conversely, thelimitations of the NERP step 1 were that it was poorly integrated
in the wider policy and quality improvement agenda of HSE.
Moreover, the lack of specification of the sociotechnicalelements of the scalable unit created challenges incommunicating the scope of the program to key stakeholders
and restricted the ability of program managers and implementers
to test and refine the change package. Regarding design, thetheory-driven analysis of the NERP step 1 highlighted that it isnecessary to specify the sociotechnical elements of a national
health IT program at the national level. Of note, these do notneed to be specified in tandem with technical elements but doneed to be specified quite early in the implementation process,so that the change package can be tested and refined as a set ofinterventions for implementing the scalable unit (technical andsociotechnical elements). These principal results are detailed
and compared with prior work in the next section.

Comparison with Prior WorkStrengths and Limitations of the Scale-Up of the NERP
Step 1
The first research question posed by this paper asked what are
the strengths and limitations of the scale-up of the NERP step
1 as a technical-only intervention. A key strength of the NERP
step 1 is that it scaled-up the technical capability of GPs to
submit electronic referral requests to at least one OPD specialty
in all public hospitals. The four patient safety improvements
reported by the NERP step 1 participants, including speed oftransfer, more complete demographic information, legibility,
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

McGovern et al
and traceability, have been recognized internationally as key
benefits of implementing an electronic referral solution [3,26].
Second, the NERP step 1 has maintained progress in
implementing a national electronic referral solution between
GPs and hospital OPDs beyond the pilot stage. This commitment
to enact learnings from a pilot stage must be commended sinceeHealth initiatives have been described as “plagued by ‘pilotitis’,with many small initiatives sprouting without any real
coordination or ability to scale” [27]. Specifically, in reference
to electronic referral systems, Bouamrane and Mair havereported that deployment is often “slow and characterized bylimited and localized uptake, or regional rather than nation-wide
implementations” [1]. Driven by executive leadership [5] within
eHealth Ireland, as well as GPs’ appetite and technologicalcapability for electronic referrals, the NERP step 1 has gained
a considerable foothold at a national scale. Latest figures fromFebruary 2018, for instance, indicate that 16,752 electronic
referral requests were submitted by GPs to hospital OPDs
(Multimedia Appendix 2), representing 22.5% of the overall
number of electronic referral requests submitted to Irish hospital
OPDs in that month (Multimedia Appendix 3) [28]. The third
key strength of the NERP step 1 is that it was perceived bynational-level participants as building “credibility” within thehealth system for a newly established eHealth Ireland unit. Thisfinding is important because it supports Eason’s suggestion thatwhen scaling national health IT programs, there are “manyagencies involved in shaping the system that reaches the users.
Each agency can be considered a locus for part of the decision
making” [13]. That is, agencies like eHealth Ireland set thestrategic priorities in the program design, but crucially, they
then need to have the credibility to successfully engage
implementation sites (GPs and hospital OPDs) to adopt thetechnology. Huang et al flagged that the implementation ofnational health IT programs can just as easily fail at thisinstitutional level as it can if the technology is not accepted by
the end users [29]. This credibility is particularly important in
the context of participant references to a “legacy belief that HSE
ICT projects fail.”
Associated with this institutional complexity, a key limitationof the NERP step 1 was that it was poorly integrated within thewider policy and quality improvement agenda of the healthservice. The program was designed and implemented by eHealth
Ireland to achieve a technical objective. National-levelparticipants described the program as a separate piece of work
to the HSE’s OSPIP, which is responsible for service
improvement more generally within hospital OPDs. Greenhalgh
et al cautioned that postponing the collaborative,
cross-institutional work needed to deliver a sociotechnicalimplementation only increases the chances that a technicalsystem will be met with resistance from other stakeholders [26],particularly if the aims of the program do not align with theprofessional norms of the end users [30]. In addition, Huang etal suggested that major health sector innovations typically“emerge from negotiations between diverse stakeholders whocompete to impose or at least prioritize their preferred versionof that innovation” [29]. Instead of a top-down approach totechnology deployment, Coiera advocated for a “middle-out”
approach to developing national health IT systems, whereby
technical goals are set to help achieve clinical or service
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.83
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
standards [31]. These standards are not static, and therefore, apartnership approach is required between health care providers(clinicians and managers) and government and the IT industry
to constantly develop national health IT systems in line withhealth service priorities and the evolving potential of technology.
Under this approach, “implementation never stops” [31] andimplementing technical capability as an objective separate froma specific clinical or service target would not be pursued.
The second, related limitation of the NERP step 1 was theincomplete specification of the program’s scalable unit. Once
all hospitals had at least one OPD specialty accepting electronic
referral requests from GPs, the single objective for step 1 of the
program was achieved. In this study, participants reported an
implementer burden owing to this lack of specification. Hospital
participants reported having to try and figure out for themselves
how to engage local GPs, although they did not consider thistheir responsibility. Some hospital administrators expressed
dissatisfaction with the low level of consultation, which inhibitedthem from communicating the “double-jobbing” challenges
associated with sending the GP triage outcome message. Sending
this message was a feature of the pilot project but not the NERP
step 1; however, interviews with hospital administrators
suggested that the scope of the NERP step 1 was neither clearlyspecified nor communicated to them. This limited program
specification also restricted the potential to develop datacollection and reporting systems, through which individual
implementation sites could monitor their progress [32]. If the
program had been better integrated within the wider quality
improvement agenda in the HSE, a broader range of mandatory
clinical or service targets could have been set, as was the casein the rollout of electronic referrals in Scotland [1]. Importantly,
as described by participants, clinicians do not wish to work to“political targets.” Any additional targets set must be
patient-centered to ensure that the learnings gained from thedata are meaningful for various stakeholders [26,33].

Going to Full Scale With a National Health IT Program
The second research question asked whether the sociotechnicalelements of a large-scale national health IT program need to be
specified at the national policy level. Analyzing the NERP step
1 using the Framework for Going to Full Scale [16] revealed
that the scalable unit for the NERP step 1 did not include aminimum specification of the sociotechnical elements, andcritically, this incomplete specification of the scalable unit
inhibits the scale-up of the NERP step 1. The one element
specified for the NERP step 1 was that all publicly funded Irish
hospitals would be set-up with the technical capability to accept
electronic referral requests. No specification was provided forthe proportion of OPD specialties targeted, the proportion ofelectronic versus paper referral requests targeted, or the
proportion of GPs targeted to use electronic referral requests.
Consequently, data collection and reporting systems were notput in place to capture what change package was used to
implement these elements of the scalable unit at each
implementation site. For example, OPD specialties requiring a
specialist referral template instead of the standard GP-OPD
referral template remain unclear. For NERP to progress to the
Test Scale-Up phase, a more complete scalable unit needs tobe formally specified. In addition, data collection and reporting
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

McGovern et alsystems need to be put in place to monitor progress in
implementing this scalable unit, and a learning systemestablished to utilize the data collected. The learning system
synthesizes the lessons from early implementation sites to guide
future implementation sites on the design of a well-tested change
package. The answer to this second question, therefore, is yes,
the sociotechnical elements of a large-scale national health IT
program do need to be specified at the national policy level if
it is to progress to full scale. Importantly, it is the “scalable unit”(ie, goals of the program) and not the “change package” (ie, theset of interventions) that needs to be specified at the national
level, and therefore, it is not a one-size-fits-all approach to
implementation which is advocated by this study or the
Framework for Going to Full Scale. Furthermore, each
implementation site is given the flexibility to undertake a local
sociotechnical design of their “change package,” in that they
can select a set of interventions that fit the priorities andcircumstances of their local context. Ultimately, however, theirimplementation will be assessed nationally on the extent towhich the scalable unit is delivered.
The wider literature is highly critical of designing national health
IT programs with rigid top-down change packages that do notleave space for local adaption. Coiera, for instance, argued that
centrally defined, top-down implementations of national health
IT programs become increasingly out of step with service needs,
and clinical providers will have to build workarounds to makethe aging system meet emerging needs [31]. If emerging needs
are left unaddressed, the workarounds will add unmanageable
local variations to what was intended to be a singular national
design [31]. Regarding a technical-only scale-up, a dynamic
cost-benefit analysis study reported that the potential gains of
implementing an electronic message exchange could be reduced
by 40%-50%, if old working procedures to fit old technology
are maintained after new technology is implemented [34]. This
type of economic argument warrants careful consideration for
the NERP step 1 in light of the outstanding interoperability
issues between Healthlink and hospital PAS, which until
resolved will require the “double-jobbing” of old and newworking procedures. These arguments illustrate that the overall
success of the technical elements of a national health IT program
depend on their interaction and fit with the sociotechnicalsystem.
As such, the academic literature’s advocacy of a sociotechnicalapproach to implementing national health IT programs is notcontradicted by this study. This paper started with an observationthat the NERP step 1 was initiated with a technical-onlyintervention and uncertainty about whether this type of
implementation strategy [13] put the NERP step 1 at odds withthe best practice, sociotechnical approach. The key learningfrom this study is that the implementation of a national health
IT program requires an interaction and ultimately a fit between
the new technical solution and the existing sociotechnicalsystem. A program may be initiated with a technical-onlyintervention, similar to the NERP step 1; however, the priority
for such an intervention is then to fit the technical elements ofthe program to the sociotechnical system within which it isbeing implemented. It is recommended that policy makers andimplementers use a quality improvement framework such as
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.84
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Barker et al’s Framework for Going to Full Scale [16] to help
guide them in the design and implementation of national health
IT programs.

Critique of the Framework for Going to Full Scale
Barker et al’s Framework for Going to Full Scale [16] informeda theoretically driven thematic analysis of the semistructured
interview data. The concept of the “scalable unit” proved crucial
to identifying the phases of scale-up achieved by the NERP
step 1 and informing an understanding of the limitations of atechnical-only scale-up. One challenge encountered in using
this framework was that the distinction and interaction between
the “scalable unit” and the “change package” are not madeexplicit in the original paper. Both elements are described as
being generated at phase 2, Develop the Scalable Unit, with thescalable unit defined as “the smallest representative facsimileof the system targeted for full-scale implementation”; the change
package is described as “a set of context-sensitive strategies
and interventions” [16]. Having applied the framework to the
NERP step 1, it is suggested that future applications of theframework might find it helpful to think of the “scalable unit”
as the goals of a program, whereas the “change package” is theset of actions undertaken to deliver each goal. As such, bothconstructs are interdependent. As the scalable unit is developed,
the change package for implementing that scalable unit needs
to be tested and refined, and if the scalable unit changes, somust the change package. A comprehensive critique cannot beprovided on phases 3 and 4, Test Scale-Up and Go to Full Scale,because the NERP step 1 did not reach these phases. In
particular, it will be important for future applications of theframework to detail the process through which an
implementation site selects, tests, and refines their change
package (set of interventions), and at the national level, to
explore the degree of variation in the change packages employed
across implementation sites and the impact this variation has
on delivering the scalable unit.

Study Limitations
A key limitation of this study is that hospital specialists were
not interviewed. Hospital specialists were not formally engaged
in the design or implementation of the NERP step 1 andtherefore were not considered key informants in this early stageof the program. Upon the receipt of an electronic referral
request, the Hospital CRO prints the electronic referral request.
By the time it reaches a specialist, it is a paper-based referral
request, just like any other. The only change encountered by
specialists is that electronic referral requests from GPs arepresented to them for triage on a standardized template. The
triage phase of referral management was not included in the
NERP step 1, and therefore, specialist dissatisfaction or

McGovern et alsatisfaction with the standardized GP-OPD referral template
was beyond the remit of this study. Issues were raised within
the study, however, for which it would have been valuable tohave obtained a specialist medical perspective. These include
the centralization of the referral management to a CRO at a
hospital or hospital group level or the suitability of using thestandardized GP-OPD referral template for all OPD specialties.
Hence, future research should focus on these issues as they
relate to the later stages of the phases of scale-up.
The second limitation is that the participants were not recruitedfrom randomly selected implementation sites. Access to 1 pilot
site and 5 NERP step 1 implementation sites was arranged
through eHealth Ireland. Although not randomly selected, the
5 NERP step 1 sites did include public and voluntary hospitals,
as well as regional and urban hospitals, and so, a broad
representation of implementation sites was achieved.

Conclusions
This qualitative study of the early-stage implementation of the
NERP provides empirical insights into the complexity of
implementing a national health IT program. The incremental
design of this program—with step 1 only seeking to scale-up
the technical capability for the e-request phase of an electronic
referral solution—made the NERP step 1 an interesting casestudy from a sociotechnical perspective.
The strengths of this implementation were that it successfully
scaled-up the technical capability for GPs to submit electronic
referral requests to at least one specialty in all hospitals in the
Irish public health system. In addition, it maintained progressin the implementation of an electronic referral solution beyond
piloting despite limited resources and outstanding
interoperability issues. Finally, it built credibility and confidence
in the new eHealth Ireland unit’s ability to successfully
implement a national health IT program. Conversely, thelimitations of this program were that it was poorly integrated
within the wider quality improvement agenda of the HSE. The
incomplete specification of the vision for full scale createduncertainty for stakeholders on their roles and responsibilities
within the program, as well as a lack of clarity on the emerging
change packages, which need to be tested and refined.
These limitations were a consequence of not specifying acomplete scalable unit, including the sociotechnical elementsof the program. In conclusion, although the sociotechnicalelements of a program do not have to be specified in tandemwith technical elements, they do need to be specified quite earlyin the implementation process so that the potential change
packages for implementing the scalable unit can be tested andrefined into a scalable set of interventions.

Acknowledgments
The authors wish to sincerely thank the 41 key stakeholders who kindly gave their time to participate in this study. We also thankeHealth Ireland for their assistance in recruitment. This study was conducted by Applied Research for Connected Health, atechnology center led by University College Dublin and funded by Enterprise Ireland and IDA Ireland.

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.85
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

Conflicts of Interest
None declared.

Multimedia Appendix 1
Number of electronic referral requests submitted to participating hospital outpatient departments in 2015/2016. Data collection
period is Oct 2015-May 2016. Source: eHealth Ireland.
[PNG File, 48KB - medinform_v6i3e10488_app1.png ]Multimedia Appendix 2
Monthly number of electronic referral requests (Sep 2015-Feb 2018). Source: eHealth Ireland.
[PNG File, 67KB - medinform_v6i3e10488_app2.png ]Multimedia Appendix 3
Upward trend in % of general electronic referrals to outpatient departments (Mar 2016-Feb 2018). Source: eHealth Ireland.
[PNG File, 48KB - medinform_v6i3e10488_app3.png ]References
1.

2.
3.

4.
5.

6.

7.

8.
9.

10.
11.

12.

13.
14.

Bouamrane M, Mair FS. Implementation of an integrated preoperative care pathway and regional electronic clinical portal
for preoperative assessment. BMC Med Inform Decis Mak 2014 Nov 19;14:93 [FREE Full text] [doi:10.1186/1472-6947-14-93] [Medline: 25407812]
Mehrotra A, Forrest CB, Lin CY. Dropping the baton: specialty referrals in the United States. Milbank Q 2011
Mar;89(1):39-68 [FREE Full text] [doi: 10.1111/j.1468-0009.2011.00619.x] [Medline: 21418312]
Gu Q, Koenig L, Faerberg J, Steinberg CR, Vaz C, Wheatley MP. The Medicare Hospital Readmissions Reduction Program:potential unintended consequences for hospitals serving vulnerable populations. Health Serv Res 2014 Jun;49(3):818-837[FREE Full text] [doi: 10.1111/1475-6773.12150] [Medline: 24417309]
Liddy C, Hogel M, Blazkho V, Keely E. The current state of electronic consultation and electronic referral systems in
Canada: an environmental scan. Stud Health Technol Inform 2015;209:75-83. [Medline: 25980708]
Tuot DS, Leeds K, Murphy EJ, Sarkar U, Lyles CR, Mekonnen T, et al. Facilitators and barriers to implementing electronic
referral and/or consultation systems: a qualitative study of 16 health organizations. BMC Health Serv Res 2015 Dec
19;15:568 [FREE Full text] [doi: 10.1186/s12913-015-1233-1] [Medline: 26687507]
Hayes M. Health Service Executive. 2010. Report of the Review of Radiology Reporting and the Management of GP
Referral Letters at Adelaide and Meath Hospital (Dublin), incorporating the National Children’s Hospital (AMNCH),
Tallaght Hospital URL: https://hse.ie/eng/services/publications/hospitals/Tallaght%20Hospital%20Review%202010.pdf[WebCite Cache ID 6y6t03jid]
Health Information and Quality Authority (HIQA). Report and Recommendations on Patient Referrals from General Practiceto Outpatient and Radiology Services, including the National Standard for Patient Referral Information. 2011. Health
Information and Quality Authority URL: https://www.hiqa.ie/reports-and-publications/health-information/
report-and-recommendations-patient-referrals-general [WebCite Cache ID 6y6teTPMS]
Wootton R, Harno K, Reponen J. Organizational aspects of e-referrals. J Telemed Telecare 2003;9 Suppl 2:S76-S79. [doi:10.1258/135763303322596354] [Medline: 14728770]
Naughton E. Health Service Executive. Electronic General GP Referrals: Case Study URL: http://www.ehealthireland.ie/
Library/Document-Library/Case-Studies/Electronic-GP-Referral-Case-Study.pdf [accessed 2018-03-22] [WebCite Cache
ID 6y6uxcLxM]
Heimly V. Electronic Referrals in Healthcare: A Review. In: Medical Informatics in a United and Healthy Europe. IOS
Press 2009:327-331.
Tian L. Semantic Scholar. 2011. Improving Knowledge Management between Primary and Secondary Healthcare: an
e-Referral Project URL: https://pdfs.semanticscholar.org/8dcf/bc1c053d347538dd4c2af5a4692b0fe9b9b5.pdf [accessed
2018-06-29] [WebCite Cache ID 70Wqd3l6e]
Ludwick DA, Doucette J. Adopting electronic medical records in primary care: lessons learned from health information
systems implementation experience in seven countries. Int J Med Inform 2009 Jan;78(1):22-31. [doi:10.1016/j.ijmedinf.2008.06.005] [Medline: 18644745]
Eason K. Local sociotechnical system development in the NHS National Programme for Information Technology. J Inf
Technol 2007 Jun 26;22(3):257-264. [doi: 10.1057/palgrave.jit.2000101]
Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA 2005 Mar 9;293(10):1261-1263.
[doi: 10.1001/jama.293.10.1261] [Medline: 15755949]

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.86
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
15.
16.
17.
18.

19.
20.
21.

22.

23.
24.

25.

26.
27.

28.
29.
30.

31.
32.

33.

34.

McGovern et al

Baxter G, Sommerville I. Socio-technical systems: From design methods to systems engineering. Interacting with Computers
2011 Jan;23(1):4-17. [doi: 10.1016/j.intcom.2010.07.003] [Medline: 25904163]
Barker PM, Reid A, Schall MW. A framework for scaling up health interventions: lessons from large-scale improvement
initiatives in Africa. Implementation Sci 2016 Jan 29;11(1). [doi: 10.1186/s13012-016-0374-x] [Medline: 26821910]
Silverman D. Negotiating the coresearcher mandate—service users' experiences of doing collaborative research on mental
health. In: Doing Qualitative Research. London: SAGE Publications; 2018.
O'Kelly M, Telijeur C, O'Kelly F, NiShuileabhain A, O'Dowd T. Trinity College Dublin. 2016. Structure of General Practicein Ireland 1982-2015 URL: https://www.tcd.ie/medicine/public_health_primary_care/assets/pdf/
structure-of-general-practice-2016.pdf [WebCite Cache ID 6zFxPJByw]
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101.
[doi: 10.1191/1478088706qp063oa]
Halcomb EJ, Davidson PM. Is verbatim transcription of interview data always necessary? Appl Nurs Res 2006
Feb;19(1):38-42. [doi: 10.1016/j.apnr.2005.06.001] [Medline: 16455440]
Higgins JR. Department of Health. 2013. The Establishment of Hospital Groups as a transition to Independent Hospital
Trusts URL: http://health.gov.ie/wp-content/uploads/2014/03/IndHospTrusts.pdf [accessed 2018-03-22] [WebCite Cache
ID 6y72EIOCI]
Health Service Executive. 2014. Community Healthcare Organisations – Report and Recommendations of the Integrated
Service Area Review Group URL: https://www.hse.ie/eng/services/publications/corporate/choreport.html [WebCite Cache
ID 6y72RLdbx]
Harris S. Houses of the Oireachtas Open Data APIs. 2017. Committee on the Future of Healthcare Opening Statement
Minister for Health Simon Harris T.D URL: https://tinyurl.com/ycabprka [WebCite Cache ID 6y72tSZEd]
Health Service Executive. 2016. Strategy for the Design of Integrated Outpatient Services 2016-2020 URL: https://www.
hse.ie/eng/services/list/3/acutehospitals/patientcare/outpatientservices/strategy-for-the-design-of-integrated-outpatient-service.
pdf [WebCite Cache ID 6y73Bg3wT]
Plunkett O, Burke B, Hegarty I, Nugent I, O'Shaughnessy C. Outpatient Performance Improvement Programme 2014-2015.
2013. Protocol: The Management of Outpatient Services URL: http://www.olchc.ie/Children-Family/Outpatients/
Protocol-for-the-Management-of-Outpatient-Services-Ed1-Jan-2013.pdf [accessed 2018-03-22] [WebCite Cache ID
6y73LFvJe]
Greenhalgh T, Stones R, Swinglehurst D. Choose and Book: a sociological analysis of ‘resistance’ to an expert system.
Soc Sci Med 2014 Mar;104:210-219 [FREE Full text] [doi: 10.1016/j.socscimed.2013.12.014] [Medline: 24581080]
Godaro B, Nikore V. What are Health Informatics, eHealth and mHealth? In: Celi LAG, Fraser HSF, Nikore V, Osorio JS,
Paik K, editors. Global Health Informatics: Principles of eHealth and mHealth to Improve Quality of Care. Cambridge,
Massachusetts: The MIT Press; 2017:77-91.
eHealth Ireland. eReferral: Latest Data URL: http://www.ehealthireland.ie/Strategic-Programmes/eReferral/Latest%20Data/
Latest-Data.html [accessed 2018-05-08] [WebCite Cache ID 6zG9RLkhG]
Huang F, Blaschke S, Lucas H. Beyond pilotitis: taking digital health interventions to the national level in China andUganda. Global Health 2017 Jul 31;13(1):49 [FREE Full text] [doi: 10.1186/s12992-017-0275-z] [Medline: 28756767]
Greenhalgh T, Swinglehurst D, Stones R. Rethinking resistance to ‘big IT’: a sociological study of why and when healthcare
staff do not use nationally mandated information and communication technologies. Health Serv Deliv Res 2014
Nov;2(39):1-86. [doi: 10.3310/hsdr02390] [Medline: 27466649]
Coiera E. Building a National Health IT System from the middle out. J Am Med Inform Assoc 2009 May;16(3):271-273[FREE Full text] [doi: 10.1197/jamia.M3183] [Medline: 19407078]
Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety ofhealth care: a systematic overview. PLoS Med 2011 Jan;8(1):e1000387 [FREE Full text] [doi: 10.1371/journal.pmed.1000387][Medline: 21267058]
Hartveit M, Vanhaecht K, Thorsen O, Biringer E, Haug K, Aslaksen A. Quality indicators for the referral process fromprimary to specialised mental health care: an explorative study in accordance with the RAND appropriateness method.
BMC Health Serv Res 2017 Dec 03;17(1):4 [FREE Full text] [doi: 10.1186/s12913-016-1941-1] [Medline: 28049470]
Aanesen M, Moilanen M, Olsen F. Economic gains from electronic message exchange: the importance of working procedures.
Int J Med Inform 2010 Sep;79(9):658-667. [doi: 10.1016/j.ijmedinf.2010.06.004] [Medline: 20621553]

Abbreviations
CIO: chief information officer
CRO: central referral office
GP: general practitioner
GPIT: General Practice Information Technology Group (Ireland)
HIQA: Health Information and Quality Authority (Ireland)
HSE: Health Service Executive (Ireland)
http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.87
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

McGovern et al

ICGP: Irish College of General Practitioners (Ireland)
ICT: information communication technology
iPMS: Integrated Patient Management System
IPMS: Integrated Patient Management System Programme (Ireland)
IT: information technology
NTPF: National Treatment Purchase Fund (Ireland)
NERP: National Electronic Referral Programme (Ireland)
OPD: outpatient department
OSPIP: Outpatient Services Performance Improvement Programme (Ireland)
PAS: patient administrative system
PDSA: Plan-Do-Study-Act cycle of quality improvement
UCD: University College Dublin

Edited by G Eysenbach; submitted 23.03.18; peer-reviewed by K Eason, C Liddy, H Yu; comments to author 12.04.18; revised version
received 23.05.18; accepted 24.05.18; published 18.07.18.
Please cite as:
McGovern M, Quinlan M, Doyle G, Moore G, Geiger S
Implementing a National Electronic Referral Program: Qualitative StudyJMIR Med Inform 2018;6(3):e10488
URL: http://medinform.jmir.org/2018/3/e10488/
doi:10.2196/10488
PMID:30021709©Marcella McGovern, Maria Quinlan, Gerardine Doyle, Gemma Moore, Susi Geiger. Originally published in JMIR Medical
Informatics (http://medinform.jmir.org), 18.07.2018. This is an open-access article distributed under the terms of the Creative
Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The
complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright andlicense information must be included.

http://medinform.jmir.org/2018/3/e10488/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10488 | p.88
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Brajovic et al

Original Paper

Validating a Framework for Coding Patient-Reported Health
Information to the Medical Dictionary for Regulatory Activities
Terminology: An Evaluative Study
Sonja Brajovic1, MD; David A Blaser2, PharmD; Meaghan Zisk2, RN, MPH; Christine Caligtan2, RN, MSN; Sally
Okun2, RN, MMHS; Marni Hall3, PhD, MPH; Carol A Pamer1, RPh, MSc
1

Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, US Food And Drug Administration, Silver Spring, MD, UnitedStates2

PatientsLikeMe, Cambridge, MA, United States

3

Center for Advanced Evidence Generation Real World Insights, IQVIA, Rockville, MD, United States

Corresponding Author:
David A Blaser, PharmD
PatientsLikeMe
160 Second Street
Cambridge, MA, 02142
United States
Phone: 1 781 718 6382
Fax: 1 866 850 6240
Email: dblaser@patientslikeme.com

Abstract
Background: The availability of and interest in patient-generated health data (PGHD) have grown steadily. Patients describe
medical experiences differently compared with how clinicians or researchers would describe their observations of those same
experiences. Patients may find nonserious, known adverse drug events (ADEs) to be an ongoing concern, which impacts thetolerability and adherence. Clinicians must be vigilant for medically serious, potentially fatal ADEs. Having both perspectives
provides patients and clinicians with a complete picture of what to expect from drug therapies. Multiple initiatives seek toincorporate patients’ perspectives into drug development, including PGHD exploration for pharmacovigilance. The Food andDrug Administration (FDA) Adverse Event Reporting System contains case reports of postmarketing ADEs. To facilitate theanalysis of these case reports, case details are coded using the Medical Dictionary for Regulatory Activities (MedDRA).
PatientsLikeMe is a Web-based network where patients report, track, share, and discuss their health information. PatientsLikeMe
captures PGHD through free-text and structured data fields. PatientsLikeMe structured data are coded to multiple medical
terminologies, including MedDRA. The standardization of PatientsLikeMe PGHD enables electronic accessibility and enhances
patient engagement.
Objective: The aim of this study is to retrospectively review PGHD for symptoms and ADEs entered by patients on
PatientsLikeMe and coded by PatientsLikeMe to MedDRA terminology for concordance with regulatory-focused coding practices.
Methods: An FDA MedDRA coding expert retrospectively reviewed a data file containing verbatim patient-reported symptoms
and ADEs and PatientsLikeMe-assigned MedDRA terms to determine the medical accuracy and appropriateness of the selected
MedDRA terms, applying the International Council for Harmonisation MedDRA Term Selection: Points to Consider (MTS:PTC)
guides.
Results: The FDA MedDRA coding expert reviewed 3234 PatientsLikeMe-assigned MedDRA codes and patient-reported
verbatim text. The FDA and PatientsLikeMe were concordant at 97.09% (3140/3234) of the PatientsLikeMe-assigned MedDRA
codes. The 2.91% (94/3234) discordant subset was analyzed to identify reasons for differences. Coding differences were attributed
to several reasons but mostly driven by PatientsLikeMe’s approach of assigning a more general MedDRA term to enable
patient-to-patient engagement, while the FDA assigned a more specific medically relevant term.
Conclusions: PatientsLikeMe MedDRA coding of PGHD was generally comparable to how the FDA would code similar data,
applying the MTS:PTC principles. Discordant coding resulted from several reasons but mostly reflected a difference in purpose.
The MTS:PTC coding principles aim to capture the most specific reported information about an ADE, whereas PatientsLikeMe
may code patient-reported symptoms and ADEs to more general MedDRA terms to support patient engagement among a larger
http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.89
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Brajovic et algroup of patients. This study demonstrates that most verbatim reports of symptoms and ADEs collected by a PGHD source, suchas the PatientsLikeMe platform, could be reliably coded to MedDRA terminology by applying the MTS:PTC guide. Regarding
all secondary use of novel data, understanding coding and standardization principles applied to these data types are important.
(JMIR Med Inform 2018;6(3):e42) doi:10.2196/medinform.9878
KEYWORDS
adverse drug events; Food and Drug Administration; MedDRA; patient-generated health data; PatientsLikeMe; vocabulary,
controlled; data curation

Introduction
Patients describe their medical experiences differently comparedwith how clinicians or researchers would describe theirobservations. The interpretation of data gathered from patientsis typically based on a clinician’s perspective of how a patientis feeling or functioning or what is of most concern to aclinician. However, a clinician’s impression may differ from apatient’s experience and therefore may not be an accurate orcomplete interpretation [1-3]. For example, a patient may finda nonserious, known adverse drug event (ADE) to be an
ongoing, daily concern, which impacts tolerability andadherence. Clinicians, however, must be vigilant for medicallyserious or potentially fatal ADEs. Having both perspectives canprovide patients and clinicians with a complete picture of what
to expect from drug therapy options for a given medical
condition.
With new advances in technology, the availability of and interestin patient-generated health data (PGHD) have grown steadily.
PGHD are distinct from data generated in clinical settings, suchas within clinical trials or during encounters with health carepractitioners. Most importantly, patients directly report andrecord these data and are responsible for sharing and distributing
them [4]. The internet democratized access to and sharing ofhealth information, leading to the creation of Web-based patient
networks. The growing popularity and advancement of mobile
sensors, wearable devices, and smartphones have createdenormous opportunity for innovative approaches tounderstanding, managing, and improving health through thecollection and application of PGHD [5]. Several ongoing
initiatives incorporate patients’ perspectives into drug
development processes. For example, the Patient-Centered
Outcomes Research Institute was established to address
challenges with traditional research and integrate patients as
key stakeholders and partners in the research process [6]. In
addition, the US Food and Drug Administration (FDA)
Prescription Drug User Fee Act V mandated public engagement
through patient-focused drug development workshops anddevelopment of patient-reported outcomes to better understandthe impact of symptoms and treatments on patients’ lives [7].
The 21st Century Cures Act, among other provisions, also
requires the FDA to consider real-world evidence andpatient-experience data in its review of drugs and devices [8].
The FDA Adverse Event Reporting System (FAERS) contains
case reports of postmarketing adverse events submitted to the
FDA Center for Drug Evaluation and Research and Center for
Biologics Evaluation and Research [9]. The FDA uses thesereports to generate and evaluate signals of potential adverse
http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

reactions to drugs and biologics [10]. To facilitate searching
and analysis of the case reports in FAERS, all suspected ADEs,
such as medical diagnoses and stand-alone signs or symptoms,
as well as medication errors and product quality issues included
in case narratives, are coded using the Medical Dictionary forRegulatory Activities (MedDRA). MedDRA is a terminology
endorsed by the International Council on Harmonisation (ICH)to be used by regulators and the pharmaceutical industry to
codify ADEs reported with the use of medical products [11].
The ICH MedDRA Term Selection: Points to Consider(MTS:PTC) document provides valuable guidance on bestpractices for term selection and promotes accuracy andconsistency in coding [12]. Pharmaceutical companies precode
the reported suspected adverse reactions to drugs or biologics
to MedDRA according to the MTS:PTC guide. The FDA thensamples and reviews the submitted MedDRA codes for coding
quality, medical accuracy, and alignment with MTS:PTC.
Moreover, the FDA applies the MTS:PTC guide to internallycode FAERS case reports received directly through MedWatch,
the FDA’s voluntary reporting system for health care
professionals, consumers, and patients [13]. Medically accurate
and thorough MedDRA coding is an essential prerequisite forsubsequent reliable and comprehensive retrieval of pertinent
cases through electronic querying by MedDRA codes. While
MedDRA terminology is critical for searching databases andconducting data mining activities, these terms do not necessarily
fully reflect a patient’s experience.
PatientsLikeMe (PLM) is a Web-based network where patients
report, track, share, and discuss their health information. PLM
captures PGHD through free-text entries and structured data
fields; these data have previously been used to conduct researchfrom patients’ perspectives [14-16]. The structured fields forentering data, such as patients’ conditions, symptoms, or
treatments, are coded by PLM to several established medical
terminology systems, including MedDRA, to increase its
usefulness for research and enable interoperability with external
systems.
This study aimed to evaluate the concordance and discordanceof MedDRA coding results of verbatim patient reports in a
structured PGHD setting compared with how the same
submissions would be coded based on the MTS:PTC coding
guide, the current standard for regulatory settings.

Methods
We conducted a retrospective evaluation of PLM MedDRA
coding of PGHD of signs, symptoms, and ADEs collected instructured data fields on the PLM platform and compared thisJMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.90
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
with how the same PGHD data would be coded to MedDRA
applying the MTS:PTC. Medical terminology systems, such as
MedDRA, are typically developed to be used by health care
professionals, regulators, pharmaceutical industry, andresearchers. Although MedDRA contains “lay-person” terms,
it does not contain all the language familiar to most patients.
To help bridge the gap between clinical terminology in
MedDRA and patient-friendly language, the PLM terminology
was created using the words and phrases reported by patients.
The PLM terminology, which is maintained and curated by ateam of medical professionals, consists of several types of
medical entities, each of which is coded to one or more external
medical terminology systems (Table 1).
Patients who wish to share and track their personal health history
can search the PLM terminology to find an item (eg, condition,
symptom, and treatment) to add to their profile. As patientsenter text in the search field, a range of possible matches in the
PLM terminology are suggested; patients may choose anappropriate match or submit a request to add a new item. A
medical professional at PLM reviews the request and determines
whether it can be merged to an existing item in the PLM
terminology (Figure 1).
When a new item is added to the PLM terminology, a detailed
entry is created to describe and distinguish the concept
appropriately (Figure 1). For each new item, PLM writes anappropriate patient-facing description, codes the item toappropriate medical terminologies, and sets any relevant
customizations to guide data entry for future patients. Once thisprocess is complete, the curated PGHD becomes a new item in
the PLM terminology and appears on the patient profile. Finally,
PLM sends a message to notify the patient that the new item
has been added to the PLM terminology. When a verbatim
patient report is curated for any reason, the patient receives amessage alerting them to the change and providing them withthe option to accept or reject the change and to request theoriginal entry be retained.
A data file was generated containing PGHD for symptoms and
ADEs entered through the PLM platform from January 1, 2013
to September 1, 2015; this file contained the patient’s verbatim
text, the item(s) it was merged to in the PLM terminology, andthe MedDRA term associated with the item(s) as assigned by
PLM. Only PGHD from patients with an active account from

Brajovic et al
the United States were included in the data file. Notably, thedata file did not include any patient-level, personally identifiable
information. A MedDRA terminology expert reviewer from the
FDA evaluated each record in the file for accuracy andspecificity of the MedDRA terminology coding, applying the
MTS:PTC guide. MedDRA has a hierarchical structure with 5
levels as follows: System Organ Class (SOC), High-Level Group
Term (HGLT), High-Level Term (HLT), Preferred Term (PT),and Lowest-Level Term (LLT). The PT level is considered a
distinct descriptor (single medical concept) [10]. The LLTs
under a PT are synonyms and lexical variants of the PT; each
LLT is linked to only one PT. LLTs reflect how information
might be reported and represent the coding level. Symptoms
and ADEs in the PLM database are coded to an LLT, as per theestablished coding practice.
In this study, the FDA reviewer evaluated the appropriatenessof the MedDRA term(s) associated with the patients’ verbatim
text. The FDA reviewer provided written comments for eachrecord indicating one of the following coding results categories:
•

•
•
•

•

•

MedDRA coding appropriate: Both the FDA and PLM
reviewer agreed that the selected MedDRA term wasappropriate for the PGHD verbatim.
Incorrect MedDRA coding: The MedDRA term assignedby PLM was not appropriate.
Missed concept: A component of the PGHD verbatim entry
was not coded to a MedDRA term.
Outdated MedDRA version: Reviewer recommended amore specific MedDRA term, but it was not available in
the MedDRA version used by PLM.
Duplicate: A unique record was listed more than once inthe extracted dataset. These records were not included in
the final analysis.
Unable to assess: PLM’s coding decision for these records
was based on additional information available as free text
on the patient’s profile and private message communication
between PLM and the patient. Specifically, PLM engages
with patients through private messages to learn more about
patients’ experiences to capture the information as reported
accurately; this additional information was not available inthe data file provided to the FDA for review. Thus, theserecords could not be properly evaluated for coding accuracy
and were excluded from the final analysis.

Table 1. Coding for data elements of the PatientsLikeMe terminology.
Medical entity type

ExamplesCondition

Multiple sclerosis, epilepsy, traumatic brain injury, major depressive Systematized Nomenclature of Medicine-ClinicaldisorderTerms (SNOMED CT), International Classificationof Diseases-10 (ICD10), Medical Dictionary forRegulatory Activities (MedDRA)

Symptom

Depressed mood, anxious mood, fatigue, insomnia, pain

SNOMED CT, ICD10, MedDRA, International
Classification of Functioning, Disability and Health

Treatment

Gabapentin, vitamin D, physical therapy, cognitive behavioral
therapy

RxTerms

Side effect

May be coded to a condition or symptom

See above for Terminology

Treatment purpose

May be coded to a condition or symptom

See above for Terminology

Hospitalization reason

May be coded to a condition, symptom, or treatment

See above for Terminology

http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

Terminology

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.91
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Brajovic et al

Figure 1. Diagram of information flow for patient submissions. MedDRA: Medical Dictionary for Regulatory Activities; PLM: PatientsLikeMe.

For each included record, other than those classified as“MedDRA coding appropriate,” the reviewer provided arationale for disagreement and proposed an alternative MedDRA
code. Together, the FDA and PLM reviewers discussed each
discordant record, reached consensus on the category, anddocumented the reasons for discordant coding.

Results
The data file from PLM contained 3349 submissions. A totalof 115 items were identified as “duplicates” (n=20) or “unable
to assess” (n=95) and were excluded from the final analysis.
Examples of entries in this category are: “loss of blood,” LLT
Rectal bleeding (patient clarified the source of blood loss);“systemic nerve overstimulation,” LLT Essential tremor (patientclarified); “blood pressure,” LLT Blood pressure high (patientreported high blood pressure just as “blood pressure”); “hosehurts my ears,” LLT Skin irritation [a patient with chronic
obstructive pulmonary disorder on supplemental oxygentherapy].
A total of 3234 MedDRA-coded items remained for the final
analysis. The expert reviewer determined that these verbatim
terms could be MedDRA-coded as per the patient-reported
submission and found that MedDRA coding was appropriate
in 97.09% (3140/3234) of cases (Table 2). These reported
verbatim terms ranged from specific medical terms (“intercostal
neuralgia” and “coccydynia”), to less specific terms (“balance

problems” and “blood clots in legs”) and personalcommunications (“just didn’t feel right” and “zombie mommy”).
Table 3 illustrates examples of differences between the PLM
and FDA approach to coding based on MTS:PTC.
In Table 3, example 1 wherein the patient submission of“Pruritis” was merged into the existing PLM term “Itching,”
demonstrates an instance in which more patient-friendly
language is preferred by PLM. In the PLM terminology, theterm “itching” encompasses all entries related to “pruritus,”“itch,” “itchy skin,” and “itchiness.” After discussion, the FDA
reviewer agreed that the coding was appropriate at the PT level,
as both LLT Pruritus and LLT Itching roll up to the same PT
Pruritus. This example illustrates that the language used bypatients can range from informal to highly technical.
Example 2 is an instance of PLM coding an item to more
specific MedDRA terms than the FDA. The PLM terminology
contains separate terms for a symptom affecting different bodyarea; this approach enables patients to track and monitor asymptom on each body area individually. Thus, PLM chose to
split the patient submission “rash on chest & neck” into theindividual symptoms “rash on chest” and “rash on neck.” “Rashon chest” was mapped to the LLT Skin rash, whereas “rash onneck” was coded to the LLT Neck rash. However, the FDA
coding practice is to code “rash on chest and neck” to a single
MedDRA term, LLT Rash, and consider it a single event as theterm for both “rash on chest” and “rash on neck” roll up to thesame PT in the MedDRA hierarchy.

Table 2. Results of the Food and Drug Administration Medical Dictionary for Regulatory Activities (MedDRA) coding review (N=3234a).

aCategory

Records, n (%)

Coding appropriate

3140 (97.09)

Incorrect coding

45 (1.39)

Missed concept

38 (1.18)

Outdated MedDRA version

11 (0.34)

Excludes “duplicates” (n=20) and “unable to assess” (n=95).

http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.92
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Brajovic et al

Table 3. Examples of Medical Dictionary for Regulatory Activities (MedDRA) coding from the analysis.
#

Patient submission

PLMa action

LLTa

FDAb reviewer assessment

OutcomePruritus

Coding appropriate

N/Ae

Coding appropriate

N/A

PLM MedDRA codingcItchingPTd1

PruritisMerged into PLM
symptom “Itching”

2

Rash on chest &neckMerged into PLM
Skin rash; neck
symptoms “Rash on rash
chest” and “Rash on
neck”

Rash

3

Drop foot fromsurgery

Merged into PLM
symptom “Footdrop”

Foot drop

Peroneal nerve pal- Coding appropriate; however,symore specific term is available.
Recommend: LLT Peroneal nerve
palsy postoperative; PT Peroneal
nerve palsy postoperative

4

Sexual dysfunction/no libido

Merged into PLM
symptom “Sexual
dysfunction”

Sexual dysfunction Sexual dysfunction Coding appropriate

5

Irrational emotions

Merged into PLM
Mixed disturbance Antisocial behavsymptom “Irrational of conduct andior
emotions”
emotions

6

Cardiac arrhythmia

Merged into PLM
symptom “Irregularheartbeat (cardiac
arrhythmia)”

7

Increased INRf

8

Coding was notchanged because of
the PLM conceptual approach to coding to more general
term
N/A

Incorrect coding; recommend
coding to: LLT Emotional disorder; PT Emotional disorder

Coding updated

Heartbeats irregularHeart rate irregular Incorrect coding; recommend
coding to: LLT Cardiac Arrhythmia; PT Arrhythmia

Coding updated

Merged into PLM
symptom “HighINR”

INR abnormal

INR abnormal

Incorrect coding; recommend
Coding updated
coding to: LLT INR increased; PT
INR increased

I was stumbling,falling and tripping,my thinking process
was very slow andmy memory was
failing too

Merged into PLM
symptoms “Memory
problems,” “Slowed
thinking,” and“Tripping and stumbling”

Memory disturbance; slowed
thinking; muscularincoordination

Memory impairment; Bradyphrenia; coordination
abnormal

Missed concept. Report of
Coding updated to“falling” is missing; falls are an
include additional
important patient safety issue
information
which should be captured. For
stumbling and tripping, there is
LLT Stumbling and LLT Gait
tripping, both under PT Gait disturbance9

Back pain and nausea and vomiting

Merged into PLM
symptoms “Nausea
and vomiting” and“Back pain”

Vomiting; backpain

Vomiting; backpain

Missed concept (nausea)

PLM symptom“Nausea and vomiting” retired. The
separate symptoms“Nausea” and“Vomiting” remain
available

10

Gluten sensitivity

Merged into PLM
symptom “Gluten
intolerance”

Gluten intolerance Celiac disease

Outdated MedDRA version; coding appropriate for this version of
MedDRA, but more appropriate
terms available in newer versions.
Recommend coding to: LLTGluten sensitivity; PT Gluten sensitivity

Coding updated after upgrade tomore recent MedDRA version

11

Shocked by electrici- Merged into PLM
ty house
symptom “Seizures(grand mal or tonic–clonic)”

Tonic-clonicseizures

Grand mal convul- Unable to assess. Not clear from Communications
sion
information how this could be de- with patient indicattermineded this was reference to seizures

PLM: PatientsLikeMe.

b

FDA: Food and Drug Administration.

c

LLT: lowest-level term.

d

PT: preferred term.

e

N/A: not applicable.

http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.93
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Brajovic et alf

INR: international normalized ratio

Example 3 shows that in some other instances, PLM symptoms
are intentionally coded into more general MedDRA terms tohelp connect patients with similar experiences. PLM members
reporting “drop foot” are assigned the same symptom in the
PLM terminology regardless of the cause. Thus, the patient
submission “drop foot from surgery” was merged into “Footdrop,” an existing symptom in PLM terminology. “Foot drop”
was then coded to the MedDRA LLT Foot drop, which issubsumed under the PT Peroneal nerve palsy. The FDA
reviewer noted that although this was an acceptable choice forthe concept, the more specific LLT-PT Peroneal nerve palsy
postoperative is preferable because it captures the additional
information that the foot drop was consequent to a surgery.
However, all patients reporting “drop foot” are assigned thesame symptom in the PLM terminology regardless of the cause.
Example 4 demonstrates how PLM aggregates experiences
related to “sexual dysfunction,” even though this likely refers
to distinct manifestations for different genders. Patients havethe option to add more specific symptoms, which are coded tomore specific MedDRA terms when appropriate, such as“impotence” (coded to LLT Impotence) or “loss of sex drive”(coded to LLT Libido loss).
Examples 5, 6, and 7 illustrate the items in which the
PLM-assigned MedDRA terms did not align with the MTS:PTC
guide. In example 5, the selected MedDRA term was not
medically accurate for the patient submission of “irrationalemotions.” Although the LLT Mixed disturbance of conduct
and emotions appears at first to be a reasonable option, it issubsumed under the PT Antisocial behavior, which the FDA
reviewer determined was not an appropriate code. PLM agreed
and updated the coding for this symptom as per the reviewer’s
recommendation and investigated what may have contributed
to the selection of the incorrect term. The likely reason is that
only the name of the MedDRA LLT was visible during the data
entry process for adding or editing a symptom in PLM
terminology. The LLT’s association to a PT and higher levelsin the MedDRA’s hierarchy was not displayed. When adding
the new symptom “irrational emotions” to the PLM terminology,
the original PLM coder would not have likely selected the LLT
Mixed disturbance of conduct and emotions if they had beenaware of its associated MedDRA PT. Example 6 shows how 2
related medical concepts, irregular heartbeats and cardiac
arrhythmias, were considered as a single item in the PLM
terminology. PLM codes certain symptoms with both the clinicalterminology and more patient-friendly terminology to facilitate
patients’ tracking their condition and connecting with other
PLM patients with similar conditions. However, the MTS:PTC
guide states that if both a diagnosis (eg, cardiac arrhythmia)
and its characteristic signs or symptoms (eg, irregular heartbeats)
are reported, the MedDRA term for the more definitive diagnosis
should be selected. PLM has since updated its coding process
to reflect this rule. Example 7 was coded incorrectly because it
only captured the laboratory result as abnormal, rather than adirectional change as specified in the patient submission. In thiscase, PLM also updated the coding to reflect the increasedlaboratory value.
http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

Examples 8 and 9 are both instances in which a concept was
missed by PLM’s coding, although the reasons for missing theconcepts are distinct. In example 8, an additional MedDRA
term should have been selected. In example 9, the missing
concept, “nausea,” is captured in the name of the PLM symptom“nausea and vomiting,” but the assigned term (vomiting) does
not capture this additional information. As a result, the symptom
named “nausea and vomiting” was retired from patient searchin the PLM terminology; patients can now add “nausea” and“vomiting” as separate and distinct symptoms to their profile.
In example 10, the FDA reviewer identified a more appropriate
term. Further investigation of this example and other similar
instances revealed that the recommended terms were available
only in more recent MedDRA versions. Once PLM integrated
the most recent MedDRA version into the platform, this coding
was updated.
Example 11 demonstrates 1 of the 95 items that were flagged
as “unable to assess” and had been excluded from the final
evaluation. In these instances, the record available to the FDA
reviewer did not contain all the information PLM had whenassigning the MedDRA term, which was necessary to determine
if the coding was appropriate.

Discussion
Principal Findings
This examination of coding of PLM PGHD to the MedDRA
terminology revealed high concordance (3140/3234, 97.09%)
between how PLM PGHD were coded to MedDRA and how
the same submissions would be coded to MedDRA based on
the MTS:PTC guide. The remaining 2.91% (94/3234) discordant
coding instances revealed some important conceptual
differences. Namely, the coding for regulatory purposes focuseson capturing the most specific information reported to a
postmarketing safety surveillance program for ADEs andmedication errors. Coding in the PLM scenario, however,requires first coding PGHD to patient-friendly terms in the
curated PLM terminology and then coding the PLM terminology
to established medical terminologies such as MedDRA toorganize and aggregate medical information. In some instances,
PLM codes to a more generalized MedDRA term to facilitate
patients with similar symptoms or conditions in more easily
finding each other on the PLM platform and sharing theirexperiences. For example, a patient may report a specific termsuch as “petit mal seizure,” on the PLM platform, but thereported concept would be grouped under a generalized termin the PLM terminology (“seizure,” LLT Seizure), whereas thesame verbatim event (“petit mal seizure”) would be coded withspecificity in FAERS (LLT Petit mal). PGHD data, if they are
to serve the dual purposes of facilitating patient discussion andallowing for adverse event detection, need to have sufficient
flexibility in coding to achieve the former goal while retaining
the general adherence to the medical concepts that underlie
coding in the first place. Users of these data need to understandthe coding approach to optimize their data retrieval strategy.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.94
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSThis analysis reinforced the importance of several best practices.
For example, when selecting a MedDRA term, it is essential toview the LLT-PT association and preferably the entire MedDRA
hierarchy for the selected LLT to determine whether the termis, indeed, the most appropriate one. In addition, optimal coding
results are achieved when using the latest available MedDRA
version. It is important to conduct data coding quality reviewsat regularly scheduled intervals.

Limitations
Only a portion of the PLM MedDRA coding terminology wasassessed to keep the dataset to a reasonable length. In addition,
the PLM platform contains more information than the FDA
reviewer had available in the dataset. Free-text information,
which may have been private message communications about
PGHD or information on the patient profile, was not sharedwith the FDA reviewer as it could have included personallyidentifiable information. Information from other structured
sections of the profile (such as previously reported ADEs,

Brajovic et alsymptoms, and treatments) was also available to PLM during
the process of curating the PGHD but was not included in thedata file provided to the FDA reviewer.

Conclusions
This review demonstrates that PGHD consisting of signs,
symptoms, and ADE data entered by patients in curated
structured fields can be reliably coded to the MedDRA
terminology and that the coding of these data by PLM isgenerally aligned with MTS:PTC principles. Understanding thecoding purpose and approach is informative for the optimizationof data retrieval strategy. These findings suggest that efficientelectronic searching and aggregation of PGHD might be possible
when consistent, systematic curation processes are applied to
PGHD as they are reported by patients. This standardization
makes PGHD more electronically accessible and therefore
elevates the visibility and importance of events patients findmost significant.

Acknowledgments
The authors would like to thank Gerald Dal Pan, Morgan Hanger, Christian Johnson, Sanjay Sahoo, and Benjamin Heywood fortheir continued guidance and support of the FDA-PatientsLikeMe research collaboration and technical contributions to thismanuscript. We are grateful to all the members of PatientsLikeMe who have contributed their data and continue to make thisresearch possible.
The MedDRA trademark is owned by the International Federation of Pharmaceutical Manufacturers and Associations on behalfof the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human
Use.

Authors' Contributions
SB and DAB designed the research. SB, DAB, MZ, and CC conducted the research and analyzed the data. SB, DAB, MZ, andSO drafted the manuscript. CAP and MH revised it. All authors read, provided feedback, and approved the final manuscript.

Conflicts of Interest
DAB, MZ, CC, and SO are employees of PatientsLikeMe and hold stock options in the company. MH is a former employee ofPatientsLikeMe and does not hold stock options in the company. The PatientsLikeMe Research Team has received researchfunding (including conference support and consulting fees) from AbbVie, Accorda, Actelion, Alexion, Amgen, AstraZeneca,
Avanir, Biogen, Boehringer Ingelheim, Celgene, EMD, Genentech, Genzyme, Janssen, Johnson & Johnson, Merck, Neuraltus,
Novartis, Otsuka, Permobil, Pfizer, Sanofi, Shire, Takeda, Teva, and UCB. SB and CAP are employees of the US Government.
The opinions expressed in this manuscript are those of the authors and not intended to represent the opinions of the United States
Food and Drug Administration. MH reports no conflicts of interest.

References
1.
2.

3.

4.

5.

Basch E. The missing voice of patients in drug-safety reporting. N Engl J Med 2010 Mar 11;362(10):865-869 [FREE Fulltext] [doi: 10.1056/NEJMp0911494] [Medline: 20220181]
Blenkinsopp A, Wilkie P, Wang M, Routledge PA. Patient reporting of suspected adverse drug reactions: a review ofpublished literature and international experience. Br J Clin Pharmacol 2007 Feb;63(2):148-156. [doi:10.1111/j.1365-2125.2006.02746.x] [Medline: 17274788]
Hakobyan L, Haaijer-Ruskamp FM, de ZD, Dobre D, Denig P. A review of methods used in assessing non-serious adverse
drug events in observational studies among type 2 diabetes mellitus patients. Health Qual Life Outcomes 2011 Sep 29;9:83[FREE Full text] [doi: 10.1186/1477-7525-9-83] [Medline: 21958008]
Shapiro M, Johnston D, Wald J, Mon D. Patient-generated health data: white paper prepared for the Office of the National
Coordinator for Health IT.: RTI International; 2012 Apr. URL: https://www.healthit.gov/sites/default/files/
rti_pghd_whitepaper_april_2012.pdf [accessed 2018-01-22] [WebCite Cache ID 6wfQ0L8AP]
Piwek L, Ellis DA, Andrews S, Joinson A. The Rise of Consumer Health Wearables: Promises and Barriers. PLoS Med
2016 Feb;13(2):e1001953 [FREE Full text] [doi: 10.1371/journal.pmed.1001953] [Medline: 26836780]

http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.95
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
6.
7.
8.
9.

10.
11.
12.

13.
14.

15.

16.

Brajovic et al

Clancy C, Collins FS. Patient-Centered Outcomes Research Institute: the intersection of science and health care. Sci Transl
Med 2010 Jun 23;2(37):37cm18. [doi: 10.1126/scitranslmed.3001235] [Medline: 20574065]
US 112th Congress. Public Law 112–144. 2012 Jul 09. Food and Drug Administration Safety and Innovation Act URL:
https://www.congress.gov/bill/112th-congress/senate-bill/3187 [accessed 2017-08-30] [WebCite Cache ID 6wfQnPT9D]
Burwell SM. Setting value-based payment goals - HHS efforts to improve U.S. health care. N Engl J Med 2015 Mar
5;372(10):897-899. [doi: 10.1056/NEJMp1500445] [Medline: 25622024]
US Food Drug Administration. Questions and Answers on FDA's Adverse Event Reporting System (FAERS) URL: https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/default.htm [accessed2017-08-30] [WebCite Cache ID 6wfS7sC6U]
Woodcock J, Behrman RE, Dal PGJ. Role of postmarketing surveillance in contemporary medicine. Annu Rev Med
2011;62:1-10. [doi: 10.1146/annurev-med-060309-164311] [Medline: 20809798]
MedDRA: the Medical Dictionary for Regulatory Activities. MedDRA Hierarchy. URL: https://www.meddra.org/how-to-use/basics/hierarchy [accessed 2018-01-22] [WebCite Cache ID 6wfSSxVwa]
The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. ICH MedDRA
Term Selection: Points to Consider. URL: http://www.ich.org/about/mission.html [accessed 2017-08-30] [WebCite Cache
ID 71WpzKgYM]
Kessler DA. Introducing MEDWatch. A new approach to reporting medication and device adverse effects and productproblems. JAMA 1993 Jun 02;269(21):2765-2768. [Medline: 8492403]
de LLC, Dimova S, Mueller K, Phillips G, Durgin TL, Wicks P, et al. PatientsLikeMe® Online Epilepsy Community:
Patient characteristics and predictors of poor health-related quality of life. Epilepsy Behav 2016 Dec;63:20-28 [FREE Fulltext] [doi: 10.1016/j.yebeh.2016.07.035] [Medline: 27544877]
Little M, Wicks P, Vaughan T, Pentland A. Quantifying short-term dynamics of Parkinson's disease using self-reported
symptom data from an Internet social network. J Med Internet Res 2013;15(1):e20 [FREE Full text] [doi: 10.2196/jmir.2112][Medline: 23343503]
DasMahapatra P, Raja P, Gilbert J, Wicks P. Clinical trials from the patient perspective: survey in an online patientcommunity. BMC Health Serv Res 2017 Feb 27;17(1):166 [FREE Full text] [doi: 10.1186/s12913-017-2090-x] [Medline:
28241758]

Abbreviations
ADE: adverse drug event
FAERS: FDA Adverse Event Reporting System
FDA: Food and Drug Administration
HGLT: high-level group term
HLT: high-level term
ICH: International Council on Harmonisation
INR: international normalized ratio
LLT: lowest-level term
MedDRA: Medical Dictionary for Regulatory Activities
MTS:PTC: MedDRA Term Selection: Points to Consider
PGHD: patient-generated health data
PLM: PatientsLikeMe
PT: preferred term
SOC: system organ class

Edited by G Eysenbach; submitted 16.05.18; peer-reviewed by C Smith; comments to author 08.07.18; revised version received
19.07.18; accepted 19.07.18; published 21.08.18.
Please cite as:
Brajovic S, Blaser DA, Zisk M, Caligtan C, Okun S, Hall M, Pamer CA
Validating a Framework for Coding Patient-Reported Health Information to the Medical Dictionary for Regulatory Activities
Terminology: An Evaluative StudyJMIR Med Inform 2018;6(3):e42
URL: http://medinform.jmir.org/2018/3/e42/
doi:10.2196/medinform.9878
PMID:30131314

http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.96
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Brajovic et al©Sonja Brajovic, David A Blaser, Meaghan Zisk, Christine Caligtan, Sally Okun, Marni Hall, Carol A Pamer. Originally publishedin JMIR Medical Informatics (http://medinform.jmir.org), 21.08.2018. This is an open-access article distributed under the termsof the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use,
distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly
cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as thiscopyright and license information must be included.

http://medinform.jmir.org/2018/3/e42/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e42 | p.97
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

ReviewThree-Dimensional Portable Document Format (3D PDF) in
Clinical Communication and Biomedical Sciences: Systematic
Review of Applications, Tools, and Protocols
Axel Newe1,2, Dipl-Ing (FH), Dr; Linda Becker3, Dipl-Psych, Dipl-Phys, Dr
1

Chair of Medical Informatics, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany

2

NewTec GmbH, Pfaffenhofen an der Roth, Germany

3

Chair of Health Psychology, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany

Corresponding Author:
Axel Newe, Dipl-Ing (FH), Dr
Chair of Medical Informatics
Friedrich-Alexander University Erlangen-Nürnberg
Wetterkreuz 13
Erlangen, 91058
Germany
Phone: 49 913185 ext 26720
Fax: 49 91318526754
Email: axel.newe@fau.de

Abstract
Background: The Portable Document Format (PDF) is the standard file format for the communication of biomedical information
via the internet and for electronic scholarly publishing. Although PDF allows for the embedding of three-dimensional (3D) objects
and although this technology has great potential for the communication of such data, it is not broadly used by the scientificcommunity or by clinicians.
Objective: The objective of this review was to provide an overview of existing publications that apply 3D PDF technology andthe protocols and tools for the creation of model files and 3D PDFs for scholarly purposes to demonstrate the possibilities andthe ways to use this technology.
Methods: A systematic literature review was performed using PubMed and Google Scholar. Articles searched for were in
English, peer-reviewed with biomedical reference, published since 2005 in a journal or presented at a conference or scientificmeeting. Ineligible articles were removed after screening. The found literature was categorized into articles that (1) applied 3D
PDF for visualization, (2) showed ways to use 3D PDF, and (3) provided tools or protocols for the creation of 3D PDFs ornecessary models. Finally, the latter category was analyzed in detail to provide an overview of the state of the art.
Results: The search retrieved a total of 902 items. Screening identified 200 in-scope publications, 13 covering the use of 3D
PDF for medical purposes. Only one article described a clinical routine use case; all others were pure research articles. The
disciplines that were covered beside medicine were many. In most cases, either animal or human anatomies were visualized. A
method, protocol, software, library, or other tool for the creation of 3D PDFs or model files was described in 19 articles. Mostof these tools required advanced programming skills and/or the installation of further software packages. Only one software
application presented an all-in-one solution with a graphical user interface.
Conclusions: The use of 3D PDF for visualization purposes in clinical communication and in biomedical publications is still
not in common use, although both the necessary technique and suitable tools are available, and there are many arguments in favorof this technique. The potential of 3D PDF usage should be disseminated in the clinical and biomedical community. Furthermore,easy-to-use, standalone, and free-of-charge software tools for the creation of 3D PDFs should be developed.
(JMIR Med Inform 2018;6(3):e10295) doi:10.2196/10295
KEYWORDS
3D PDF; 3D visualization; interactive; clinical communication; biomedical science; tools; protocols; apps; online data sharing;scholarly publishing; electronic publishing
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.98
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Introduction
BackgroundThe best-known and most widely used data format for theexchange of electronic documents is probably the Portable
Document Format (PDF), which was standardized by theInternational Organization for Standardization (ISO) as ISO
32000-2:2017 [1]. Software that can read PDFs is installed onnearly every computer, and most internet browsers and email
clients have a built-in PDF renderer. This, in many cases, makes
this format the best means for the exchange of electronic
documents. However, the PDF offers more features than many
people are aware of. Although technically available since 2005,
a still lesser-known standard feature of the PDF is the possibility
to embed three-dimensional (3D) models, which enables theinteractive visualization (eg, zooming, panning, rotating, andselection of components) of such objects with qualified reader
software (Figure 1; for an interactive version of this figure, see
Multimedia Appendix 1).
A PDF document with embedded 3D objects (3D PDF) has highpotential in almost every scenario in which 3D objects should
be visualized and exchanged between different platforms (eg,computers with different operating systems). A highly relevant
use case is the exchange of medical and biomedical data (eg,the visualization of human anatomy for medical students [2] orclinical data such as the results of surgery planning [3,4]).
Therefore, this feature is also considered in the standard
AIIM/ASTM
BP-01-2008“Portable
Document
Format-Healthcare (PDF) A Best Practices Guide” (also known
as PDF Healthcare or PDF/H) [5,6]. This standard describes
how to use the PDF to exchange and preserve digital health careinformation in a safe and secure way.
In addition to the exchange between a few persons (eg, a doctor
and a patient), 3D PDF offers a very efficient and convenient
way for distributing 3D structures through the scientificcommunity by embedding 3D objects into scientificpublications.

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker
The primary aim of this review was to investigate thesignificance of 3D PDF in clinical communication and in
scholarly publications in the biomedical sciences. The secondary
aim of this review was to provide an overview of the technicalpossibilities and to present all currently available solutions forcreating 3D PDFs and related model files.

The Portable Document Format
The PDF is a computer file format for the platform-independent
definition of electronic documents [1]. It allows for describing
electronic documents with preserved fidelity, independently of
the software, device, and operating system used to create,
display, or print them. Furthermore, PDF is capable ofencapsulating all necessary resources (eg, texts, images, or
multimedia elements). Along with the open accessibility of thespecification, this has led to PDF being the most commonlyused file format for the exchange of electronic documents. For
example, all scholarly journals allow for an electronic
publication of their articles in PDF today.
The PDF was originally developed by Adobe Systems andevolved in the early 1990s from the PostScript page description
language for printers. The first specification of PDF was
published in 1993; since then, it has been developed further by
Adobe until version 1.7 [7], which was released in 2006. Thisversion was given to the ISO, which re-released thisspecification as ISO standard 32000-1:2008 in July 2008 [8].
The latest version (PDF 2.0) was published in July 2017 [1,9].
The PDF is now the de facto standard for the exchange ofelectronic documents. The original Adobe Reader software
alone has been distributed more than 500 million times aroundthe world [7], and countless other apps for displaying PDF
documents are available. For example, Quartz 2D, the nativegraphics rendering interface for two-dimensional (2D) graphicsof the macOS X and iOS operating systems, is based on the
PDF specification, (ie, it is an integral part of these operating
systems) [10].

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.99
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

Figure 1. Vessel tree of a liver. See Multimedia Appendix 1 for 3D PDF version of this figure and Multimedia Appendix 2 for a 3D PDF version ofthis entire article.

The Development of the Three-Dimensional Portable
Document Format
Version 1.6 of the PDF specification, published in 2004 [11],introduced the capability to embed 3D objects, such as thoseused by software for computer-aided design, into PDF files. At
the time of the introduction of this new feature, the format forthe data of 3D objects needed to be Universal 3D (U3D), a fileformat developed by the 3D Industry Forum (which Adobe waspart of) as “a single, common, and open 3D standard” and“JPEG for 3D” [12]. U3D is standardized by the European
Computer Manufacturers Association (ECMA) as ECMA-363[13]. The first software to support 3D Artwork (the PDF termfor 3D objects in PDF files) was the Adobe Reader 7 (and Adobe
Acrobat 7, respectively), published in January 2005. However,the first software that really allowed for an efficient andconvenient working with 3D models in PDF was Acrobat 3D,
released in early 2006. This version provided tools for the import
and conversion of many 3D formats and a 3D editor [14].
In late 2006, Adobe added the ability to embed 3D models in
Product Representation Compact (PRC) format into PDF (PDF
BaseVersion 1.7, ExtensionLevel 1 [15]), but this feature has
not been integrated into the ISO standard 32000-1 from 2008.
It took 11 years until the release of PDF 2.0 (32000-2:2017) inJuly 2017 to formally integrate PRC as 3D format into the PDF
standard. However, the support of editing features for 3D was
discontinued in later versions of Adobe Acrobat. Acrobat 9 Pro
Extended was the last version to include tools for conversion
and editing of 3D models. Instead, this functionality has been
outsourced to third-party software and has been sold separately
since 2010 [16].
Another feature that is notable in this context is the support of
JavaScript. This feature was introduced with PDF 1.3 in July
2000 [17] and allows for executing JavaScript commands within
a PDF document (eg, after clicking an embedded button or after
activating a 3D model).
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Relevance of Three-Dimensional Portable Document
FormatEspecially in the biomedical domain, the importance of 3D data
has grown in recent years (eg, visualization of chemicalmolecules, anatomy, or vessel systems). With the availabilityof the necessary technology for the visualization of such 3D
data, this technology should consequently be used to avoid aloss of information [18]. This is of particular relevance inscholarly publishing [19]. Furthermore, 3D PDF has a very highpotential for the exchange of medical data in clinicalcommunication [3] because it allows for in situ publishing of
3D figures [20].

Objectives
It has tremendously high potential for clinical communication,
many scholarly articles would benefit from interactive
visualization of 3D data [19], publishers encourage their authors
to use 3D PDF technology [21,22], and some journals provide
the necessary tools [23], yet it still does not seem to be common
use. Thus, the initial motivation for writing this review paper
was to test this hypothesis (ie, that using 3D PDF technology
is not common in clinical communication or in biomedical
publications). Therefore, as the primary goal, an overview was
to be created of the dissemination of 3D PDFs in biomedical
publications. The scope was not limited to purely clinical use
cases to cover as wide a range of possible apps in all fields of
medically relevant research. Furthermore, we will investigate
in which research areas and use cases 3D PDF has been applied
since the advent of this technology.
After the initial hypothesis was confirmed (200 articles were
found over a period of 10 years; see Results), we hypothesized
three possible reasons:1.

The existence and/or possibilities of 3D PDF might not bewell known among the scientific and clinical community.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.100(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
2.

3.

The necessary knowledge that would enable authors andpossible users to actually make use of 3D PDFs might be
hard to acquire.
The creation of appropriate model files and of final PDF
documents might be an overly cumbersome process.

Therefore, the secondary goal of this review was to tackle theprevious three issues by providing an overview of (1) existing
publications that apply 3D PDF technology to demonstrate its
possibilities, (2) available protocols for the creation of 3D PDFs
to provide the basic knowledge in compressed form, and (3)existing software tools for the creation of model files and 3D
PDFs to show ways to simplify the process.
Although the overview of the existing publications that apply
3D PDF technology in this field may elaborate new ideas of
how this form of visualization could augment scholarly
communication in biomedicine, the presented protocols andtools can probably be applied in other fields as well.

Methods
General Procedure
A systematic literature search was performed in the scientific
databases PubMed [24] and Google Scholar [25] to find articles
that either applied 3D PDF technology for visualization purposesor articles that generally dealt with the subject of 3D PDF in
the context of biomedical sciences.
Because PDF is a vehicle for presenting data rather than an
actual research topic, it was likely that a simple database search
for this term would not reveal comprehensive coverage of theliterature, especially for articles that simply used 3D PDF as ameans for visualization. Therefore, a search strategy with two
iterations was conceived (see detailed description subsequently).
Agreement about this strategy, the inclusion criteria, and thereview protocol (see respective sections subsequently) was
reached through discussion between all authors.
The first search was performed by AN on March 6, 2017. Three
update searches were performed by AN on September 30, 2017,
January 1, 2018, and April 30, 2018. The search results werescreened and checked for eligibility independently by AN and
LB on the basis of predefined inclusion criteria. The remaining
articles that were entered into the further analysis were reviewed
by means of a predefined review protocol by AN and LB
independently.

Newe & BeckerOR(portable
document
format)))AND(“2005”[Date-Publication]: “3000”[Date-Publication])”.
The goal of this initial search was to reveal all biomedical
publications that mentioned 3D PDFs without the most common
unwanted meanings for “PDF.” It was limited to articles
published after 2004 because 3D PDF technology was notavailable before 2005.
The results of this first iteration were then screened to exclude
articles that did not fulfill the inclusion criteria. The remainders,which will subsequently be referred to as “primary results,”
were then analyzed to identify articles that were cited as a sourceof 3D PDF creation (referred to as “tool articles;” see a moredetailed explanation subsequently). Next, both PubMed and
Google Scholar were searched for articles that cited these tool
articles to identify more articles that applied or mentioned 3D
PDF and that could not be found via the initial PubMed search.
The results of this second iteration were then also screened forthe inclusion criteria and the remaining articles (referred to as“secondary results”) went into the review process along withthe primary results and the tool articles.
Finally, a small number of additional articles that were known
to us to be eligible, but which were not discovered by thesystematic search, were included as well. Further details about
the search strategy, the search results, and the screening results
are available in Multimedia Appendix 3.

Inclusion Criteria
The general inclusion criteria for this review were defined asfollows: (1) articles from the biomedical domain that applied
3D PDF for visualization or articles that presented protocols or
tools for the creation of 3D PDFs or for the creation of 3D
PDF-specific, intermediate file formats (U3D or PRC); (2)articles in English language only; and (3) peer-reviewed articles
only.

Review Protocol
To fulfill all these objectives, all eligible articles were firstclassified into article types using four categories:
1.
2.
3.

Search Strategy
As mentioned previously, we conceived a two-tiered searchstrategy (Figure 2). The first iteration was a PubMed searchwith the following search term: “(((3d) OR (3-d) OR (three
dimensional) OR (interactive) OR (surface model)) AND ((pdf)

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

4.

Application: articles that applied 3D PDF technology forvisualization.
Descriptive: articles that described a way, method, or idea
how to use or to utilize 3D PDF technology.
Tool: articles that described a method, protocol, software,
library, or other means for the creation of 3D PDFs or model
files.
Mentioning only: articles that only mentioned the possibilityof using 3D PDF (without further details).

The assignment to more than one category (eg, descriptive andapplication) was allowed.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.101(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

Figure 2. Search strategy.

All articles of the type “application” were then further analyzed
for the 3D features used, for the availability of the 3D PDF
documents, and for the file size. The following 3D PDF features
were considered:
1.
2.
3.
4.
5.
6.

Geometry only: articles that only presented the geometryof the 3D model without further features.
Multiple colors and transparency: articles that had
transparent/colored models.
Product Manufacturing Information (PMI)/markup: articles
that used PMI features or markup for labeling objects.
Scripting: articles that used JavaScript for advanced
interactive features.
Texture: articles that used textured models.
Animation: articles that use animations.

The availability of the 3D PDFs was classified into:
1.
2.
3.

Embedded: articles that directly embedded 3D models into
the PDF version.
Supplement: articles that provided supplemental information
files with 3D PDFs.
Link to external resources: articles that provided the URL
or mail address for accessing the referred 3D PDFs.

Finally, all articles were classified into a scientific discipline(eg, “medicine,” “embryology,” or “human anatomy”; for a fulllist see Multimedia Appendix 4) and the clinically relevant
articles were analyzed further.

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Results
Search Results
The initial PubMed search retrieved 123 publications; after thethird update search, a total of 237 publications were found. Afterscreening these PubMed results, 109 articles were excluded
because “PDF” was not used as an acronym for “Portable
Document Format,” one was excluded because it was not inEnglish language, one was excluded because it did not cover abiomedical topic, and eight were excluded because they werean erratum only (ie, not a peer-reviewed article). During thefull-text eligibility assessment, 65 more articles were excluded
because they had no content relevant for 3D PDF. The initial
analysis of the remaining 53 primary results revealed 19 tool
articles, with 15 of them being duplicates (ie, four new tool
articles were identified).
The second iteration (ie, the search for articles citing the tool
articles) retrieved a total of 128 articles from PubMed and 537
articles from Google Scholar after the third update. Afterremoving 356 duplicates, 366 articles were screened. Of those,
22 were excluded because they were not in the English languageand 43 were excluded because they were not peer reviewed.
During the full-text eligibility assessment, 48 articles were
excluded because they did not fall into the biomedical domain
and 63 more articles were excluded because they had no content
relevant for 3D PDF. That resulted in 133 articles from thesecond iteration. Together with the 53 articles of the first
iteration, the four additional tool articles, and 10 articles thatwere included due to our knowledge, 200 articles were included
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.102(page number not for citation purposes)

JMIR MEDICAL INFORMATICSin the final analysis, which is shown in a modified Preferred
Reporting Items for Systematic Reviews and Meta-Analyses(PRISMA) flow diagram [26] in Figure 3.

Overall Analysis Results
Of the total 200 articles, the simple possibility of using 3D PDF(without providing further details) was mentioned in 13 articles(6.5%) [27-39]. Another 18 (9.0%) [2,3,23,40-54] articles
described a way, method, or idea how to use or to utilize 3D
PDF technology and 19 articles (9.5%) [20,40,55-71] describeda method, protocol, software, library, or other means for thecreation of 3D PDFs or model files.
In 156 (78.0%) articles, 3D PDF technology was actually applied
for the visualization of research results. In 11 (7.1%) more
articles, it was claimed that 3D PDF should be available, but
no 3D model could be found [58,72-81]. Of the 156 articles
with applied 3D PDF technology, 34 (21.8%)[40,55-57,61,64,68,82-108] had the 3D content directly
embedded into the PDF of the article, 94 (60.3%)[20,48,59,60,63,66,67,109-195] articles provided the 3D content
as supplementary material, and 28 (17.9%) [4,41,62,196-220]
referred to an external resource. The 34 articles in which the
3D objects were embedded directly into PDF of the main article
were published in 25 different journals.
Different features of 3D PDF were used. In seven articles(4.5%), only the basic geometry was displayed, but in a majorityof articles (146/156, 93.6%), at least multiple colors ortransparencies were used. More advanced features such as
PMI/markup (11/156, 7.1%), scripting (29/156, 18.6%), or

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker
textures (28/156, 17.9%) were used rarely. Only one article(0.6%) made use of the animation feature [67].
The disciplines were manifold. In most cases either animal(86/200 total article, 43.0%) or human (33/200, 16.5%)
anatomies were visualized. The other fields were general science(34/200, 17.0%), biochemistry (14/200, 7.0%), embryology(14/200, 7.0%), clinical/medicine (13/200, 6.5%), biology(3/200, 1.5%), statistics (1/200, 0.5%), bioinformatics (1/200,
0.5%), and astronomy (1/200, 0.5%).
The file size ranged between 0.2 and 429 mebibyte (MiB) (mean29.9, SD 59.9 MiB). An analysis by year of publication isprovided in Table 1; a detailed analysis for each publication isavailable in Multimedia Appendix 4.

Three-Dimensional Portable Document Format inClinical Use Cases
We found 13 publications covering the use of 3D PDF for
medical purposes. Eight of them could be assigned to the clinicalfield, whereas the other five were studies about the value of 3D
PDF as educational material for students of medicine. Only one
article described a use case from a clinical routine operation:
the distribution of results from computer-aided planning forliver surgery [3]. The evaluation of the acceptance and the user
experience in this study turned out to be very good. Thisconfirms that the usage of 3D PDF for clinical communicationin telemedicine is practicable and highly accepted by the users,
and that it has many advantages over 2D technology. The
method described there has been used for research as well [221].

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.103(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

Figure 3. Modified PRISMA flowchart.

Table 1. Number of publications dealing with 3D PDF by type (assignment to more than one article type was possible).
Year

Article type, nTotal, nTool

Application

Descriptive

Mentioning

2008

3

5

—

—

5

2009

—

6

1

—

7

2010

2

9

—

1

13

2011

3

13

1

1

18

2012

2

13

2

1

18

2013

4

20

1

2

24

2014

1

23

5

2

28

2015

1

31

2

—

34

2016

1

18

3

1

25

2017

1

12

2

4

19

2018 (until April)

1

6

—

1

9

All other publications with clinical applications of 3D PDF
reported pure research work. Day and colleagues [109]demonstrated that preoperative 3D modeling of magnetic
resonance imaging data of fistula-in-ano can reduce confusion
and ambiguity among surgeons when interpreting 3D PDF
reports instead of complex textual reports. The benefit of 3D
PDF for the simulation of surgical procedures was describedby Mavar-Haramija and colleagues [4,41] for endoscopic
endonasal interventions. Husch and colleagues [192]http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderXdemonstrated how automatically generated 3D PDF reports canbe used to visualize and assess the spatial relationships between
electrodes and brain structures and to detect misplaced
electrodes in a postoperative deep brain stimulation setting.
Finally, the findings of Elbashti and colleagues [54] emphasized
the value of 3D PDF for clinical communication with their study
on the use of 3D PDF for the documentation of maxillofacial
prostheses.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.104(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Another area where 3D PDF has proven valuable is patient
education. Both the telemedicine study on liver surgery
mentioned previously [3] and another study by Prats-Galino
and colleagues [197] pointed out that 3D PDF can make it mucheasier to convey the necessary understanding of a planned
procedure, its risks, and future consequences to a patient.
Several other studies showed that 3D PDF is a good means to
help students understand anatomy [2,27,42,198] or surgicalprocedures [199].

Interim Summary
Overall, within the investigated period of 10 years, only 200
articles were found that used or described the usage of 3D PDFs
in biomedical science, with 13 of them covering clinical topics.
Compared to the importance of 3D data in the biomedical
domain and thus to a presumed number of several thousand
publications containing 3D data in some form, this is a tiny
fraction. Therefore, we hypothesized about possible reasons forthis. We assumed that the lack of 3D PDF usage might be due
to one of the following reasons (see Introduction): (1) the
existence and/or possibilities of 3D PDF might not be wellknown among the scientific community, (2) the necessary
knowledge that would enable authors to actually make use of
3D PDF might be hard to acquire, and (3) the creation of
appropriate model files and of final PDF documents might be
an overly cumbersome process.
A verification of these hypotheses would be out of scope of thisreview. However, assuming these are true, it is possible tomitigate the causes of these hypothesized problems on the basisof the publications that were already collected: hypothesis 1
might be mitigated by the pure existence of this review ingeneral, and both hypotheses 2 and 3 might be mitigated byproviding an overview of the protocols and software for creating
PDFs that have been published so far.

Protocols and Software for Creating
Three-Dimensional Portable Document Format
Of the total of 200 articles, the tool articles (ie, articles that
described a method, protocol, software, library, or other means
for the creation of 3D PDFs or model files) accounted for 19
articles (9.5%) [20,40,55-71] (Tables 2 and 3; a combined table
is available in Multimedia Appendix 5). These tools can beclassified into two groups: (1) protocols that described allnecessary artifacts and steps for a creation workflow, and (2)software libraries or software apps that could produce 3D PDFs
or necessary intermediate file formats (U3D or PRC).
The following is a description of all these tools (software andprotocols) that have been published in the biomedical field. It
is intended to be used as a reference for authors that areinterested in using 3D PDF technology for their publications orfor applied medical use (eg, as demonstrated in [3]). However,
especially the older protocols rely on software that is no longer
available and that has no successor with the same functionality.
For a quick overview, see Tables 2 and 3, and see Multimedia
Appendix 5 for a more detailed analysis.

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker
The first protocol that described the workflow for the creationof a 3D PDF was presented by Kumar and colleagues [55] in2008 and required a minimum of four steps as well as the usageof commercial software and three intermediate file formats. It
is based mainly on Adobe Acrobat 3D, which is no longer
available today. Another very simple protocol was presented
by Barnes and Fluke [222] in 2008: along with their S2PLOT[223] library for the programming languages C, C++, andFortran, only the Adobe Acrobat 3D software was needed. The
S2PLOT library was later extended to provide additionalfeatures for creating PRC files (described subsequently). At
around the same time, Ruthensteiner and Heß [57] publishedtheir protocol for the creation of 3D PDFs of biological
specimens. It consisted of 11 steps, including four related to theprocessing of the sample itself. This protocol relied oncommercial software as well: Adobe Acrobat 3D and Adobe
3D Toolkit, both of which are no longer available today.
Approximately two years later, Kumar and colleagues [58]presented a protocol for the generation of 3D PDFs of moleculesthat required a minimum of four steps. Although this protocol
was generally based on commercial software, which in part is
no longer available, it also shows an alternative way by using
the well-known document preparation system LaTeX [224] incombination with the movie15 package [225]. However, theuse of LaTeX requires a model file in U3D format, which needs
to be created with other software.
Two months later, Ruthensteiner and colleagues [59] presented
an update of the 2008 article that included the S2PLOT libraryand that focused on virtual volume rendering in PDF documents.
This protocol included for the first time the scripting feature of
PDF. The need for commercial software, however, had still notbeen eliminated. At the beginning of 2012, de Boer andcolleagues [60] published a very detailed protocol for thecreation of 3D PDFs that described all necessary steps for abasic process and an advanced process on seven pages. It alsorelied on commercial software (Amira, Adobe Acrobat 9 Pro
Extended, Adobe Illustrator), but for the first time it allowed
for embedding custom control elements for selecting structuresof predefined views of the 3D figure. One month later, Ziegler
and colleagues [40] demonstrated how to incorporate a variety
of multimedia elements into PDF, including a textured 3D figure(an interactive model of a human face). This publication alsomentioned the possibility of using LaTeX to create the final
PDF. In August 2011, Danz and Katsaros [61] published their
protocol, which was based on a combination of free software(for model generation) and commercial software (for PDF
creation). One year later, Shin and colleagues [62] also releaseda protocol that was still purely based on commercial software;
4 months later, Phelps and colleagues [63] presented anothersolution that combined a workflow of 22 steps using commercial
and noncommercial software. Finally, Lautenschlager [64]published in 2013 another protocol that required a wide rangeof commercial and noncommercial software.

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.105(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

Table 2. Overview of requirements, restrictions, and possible output of the tools presented in the tool articles. For features, see Table 3.
ArticleSoftware needed

OutputCommercial

ProgrammingneededaFree

Articles presenting a protocol
Kumar et al [55]

Adobe Acrobat 3Db, Adobe Acrobat 3D

Model file, final PDF

No

Final PDF

Yes

Model file, final PDF

No

Model file, final PDF

No

Final PDF

No

Final PDF

NocToolkit , Adobe Photoshop, Adobe Illustrator
Barnes & Fluke [56]

Adobe Acrobat 3Db

S2PLOT

Ruthensteiner & Heß [57] Amira, Adobe Acrobat 3D Toolkitb, Adobeb

Acrobat 3D
Kumar et al [58]

Adobe Acrobat 9 Pro Extendedd, Adobe
Acrobat 3D Reviewer

LaTeX, MeshLabbde Boer et al [60]Amira, Adobe Acrobat 9 Pro Extendedd,
Adobe Illustrator

Ziegler et al [40]Adobe Acrobat 9 Pro Extendedc,dLaTeXDanz & Katsaros [61]Adobe Acrobat X Prod, Tetra 4D 3D PDF
Converter plug-in for Acrobat

MeshLab, DAZ Studio

Shin et al [62]Mimics, Autodesk Maya, Deep Explo-

No

Final PDF

No

Model file, final PDF

No

Model file, final PDF

No

Model file, final PDF

No

Model file, final PDF

No

Model file, final PDF

Yes

Amira, S2PLOT, ImageMag- Model file, final PDF
ick,

Yes

Asymptote, S2PLOT, LaTeX, libHaru

Model file, final PDF

Yes

Newe & Ganslandt [66]MevisLab

Model fileNo

Newe [68]MevisLab

Model fileNo

Newe [69]MevisLab

Model file, final PDF

No

Brandner et al [71]MevisLab

Model file, final PDF

No

rationd, Adobe Acrobat 9 Pro Extendedd
Phelps et al [63]Microsoft PowerPoint, Adobe Acrobat

MeshLabLautenschlager [64]Amira, Mimics, Adobe Acrobat 8 Prob/9

MeshLab, LaTeX, Drishtid,

d

d

d

Pro Extended , Adobe Acrobat 3D Review- SPIERSedit , Blender ,erb, Adobe Acrobat 3D Toolkitb, Tetra 4D
3D PDF Converter plug-in for Acrobat,

DAZ Studiod

GeoMagic Studioc, Abaqus FEAc, Avizoc,
VG Studio Maxc, Autodesk Mayac
Mavar-Haramija et al [65] Amira, Adobe Acrobat 9 Pro Extendedd,b

Adobe Acrobat 3D Reviewer , Tetra 4D 3D
PDF Converter plug-in for Acrobat
van de Kamp et al [67]Amira or Avizo, Cinema 4D, Deep Explorationd, Adobe Acrobat 9 Pro Extendedd

Zhang et al [70]libHaru

Articles presenting a protocol and a library
Ruthensteiner et al [59]Deep Explorationd, Adobe Acrobat 9 Pro
Extended

Barnes et al [20]dArticles presenting a software appaSolution requires the creation of individually tailored software code. Usage of LaTeX is not considered as “programming.”b

Software is no longer available.

c

Optional.

dSoftware is no longer available, but a successor which provides the same functionality is available.

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.106(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

Table 3. Overview of features of the tools presented in the tool articles (PDF: Portable Document Format).
ArticleSupported geometry type

Supported 3D PDF features

Mesh

Multiple

Polyline

Point cloud

PMI/markup Scripting

TextureAnimation

Poster image

x

x

x

x

x

x

colors
Articles presenting a protocol
Kumar et al [55]

x

x

x

x

x

x

Barnes & Fluke [56]

x

x

x

x

x

x

Ruthensteiner & Heß [57] x

x

x

x

x

x

x

Kumar et al [58]xxde Boer et al [60]

x

x

x

x

x

Ziegler et al [40]

x

x

x

x

x

Danz & Katsaros [61]xShin et al [62]xxPhelps et al [63]xxLautenschlager [64]xxMavar-Haramija et al [65] x
van de Kamp et al [67]xZhang et al [70]

x

x

x

x

x

x

x

x

x

x
x

x

x

x

Articles presenting a protocol and a library
Ruthensteiner et al [59]

x

x

x

x

x

x

x

Barnes et al [20]

x

x

x

x

x

x

x

x
x

Articles presenting a software app
Newe & Ganslandt [66]xxNewe [68]

x

x

x

x

Newe [69]

x

x

x

x

Brandner et al [71]

x

x

x

x

The first solution that could be used without any commercial
software was presented by Barnes and colleagues [20] in
September 2013 (ie, 5 years after the first appearance of 3D
PDF in biomedical publications). It was based on a number offree software tools and libraries (Asymptote [226]; S2PLOT;
LaTeX, libHaru [227]). Although this solution could be used
to create both model files and final PDFs, and it could be used
free of charge, it had a major drawback: it required writing a
program (ie, a certain set of programming skills was necessary
to apply this protocol). On the other hand, this gave the user
greater flexibility because the program code could be adapted
exactly to their needs and to the specific-use case.
Only 2 months later, two more articles with tools for the creationof 3D PDFs were published. Mavar-Haramija and colleagues[4] presented another commercial software-based protocol that
focused on control elements for a more sophisticated interaction
with the 3D model. At the same time, Newe and Ganslandt [66]presented a noncommercial solution for the creation of model
files of surface meshes in U3D format that did not require
programming skills. The final PDF could not be created withthat solution, but it reduced the number of steps for the creationof such model files and provided a convenient graphical userinterface. However, this solution required the installation of the
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderXxxthird-party[228,229].

biomedical-imaging

xframework,

MeVisLabIn September 2014, a previously unregarded feature of 3D PDF
emerged through a new protocol: animation. Van de Kamp andcolleagues [67] for the first time demonstrated the feasibility
and the usefulness of animated 3D figures in electronic
publications. Eight months later, Newe [68] presented an update
to his previous publication that provided an advanced userinterface for more flexibility. Furthermore, it provided thepossibility to embed both point clouds and line sets into theinteractive 3D figures. Previous publications usually only
considered objects that consisted of surface meshes. However,it was still limited to the creation of model files. The creationof final PDFs was not possible.
The first all-in-one solution for the creation of both model files
and the final PDF that did not require any programming was
also presented by Newe [69]. The Scientific3DFigurePDFApp
provided a graphical user interface for the assembly of 3Dmodels (meshes, line set, point clouds) and included an editor
for predefined views. It was also capable of producing
one-paged PDF files with the embedded 3D figure that could

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.107(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

directly be used as supplementary figures for scientificpublications.

existence of this review article might help to draw attention tothis technology and help to overcome this threshold.

The latest complete protocol was published by Zhang andcolleagues [70] in November 2017. Although it did not requirecommercial software, it was also based on the libHaru library(ie, it required the writing of program code). However, thispublication featured a deep insight into the U3D file format.

A further reason might be that the potential target
audience—although they know about 3D PDF in principle—do
not know how to create 3D PDFs or that they do not haveappropriate technical skills. In our systematic review, wereported the protocols and software tools that were used in the
156 articles that apply 3D technology in PDFs. Unfortunately,
many of them are based on software packages that are no longer
available. Furthermore, the usage of most of the reported
protocols or tools needed either programming skills, advanced
technical knowledge, and/or expensive, highly specialized,
commercial software. Although this gives the user great
flexibility because the program code can be adapted exactly tothe needs and to the specific-use case, we conclude that thethreshold for creating 3D PDFs is still too high and thateasy-to-use, standalone software tools are needed to facilitate
3D PDF creation. Of all the software tools analyzed in thisreview, the solution presented by Newe [69] comes closest to
this requirement because it is an all-in-one solution for thecreation of both model files and final PDFs, which is operated
via a graphical user interface. The major drawback of thissolution is the need for the software MeVisLab, which also
requires some basic training.

Finally, Brandner and colleagues [71] presented an extension
to the toolbox described in [69], which added the support of
textures.

Discussion
Summary of Main Findings
With this systematic review, we give an overview of the usageof 3D PDF in the biomedical domain with a special focus on
clinical applications. Furthermore, we investigate which
protocols and software tools for 3D PDF creation have been
published so far. In addition, we hypothesize about possible
barriers that might be responsible for the low dissemination of
3D PDF in clinical communication and biomedical sciences.
Our primary search revealed 19 tool articles that can beconsidered as the roots of 3D PDF in biomedical publishing.
For these articles, the scope has intentionally not been limited
to the biomedical domain, because many biomedical
publications seem to be inspired by the Barnes [56] publication(ie, an astronomy publication). Excluding this paper would haveleft several in-scope biomedical articles unrevealed (eg,[73,88,121,128,131]).
After the secondary search, in which we searched for articles
that cited these tool articles, 200 articles were found overall that
fulfilled the inclusion criteria and entered into further analysis.
This is a very small number with respect to the fact that in thelast 10 years (January 1, 2008 to January 1, 2018) a total numberof 9,705,959 articles were indexed by PubMed. Even if only
1% of these articles touched the subject of 3D data, 200 articles
is still a tiny fraction.
Within these 200 articles, actual 3D PDF figures were available
in 156 articles (78.0%). However, the 3D content was embedded
directly into the PDF version of the articles in 34 cases only.
One reason for this might have been that the journals did notsupport the direct embedding. This matches the experience of
the authors. Even some modern, online-only journals are notcapable (and, in some cases, not willing) to provide thepossibility to embed 3D figures directly into the PDF versionsof their articles.

Possible Reasons for the Low Dissemination ofThree-Dimensional Portable Document Format inMedicine and Biomedical Research
As mentioned previously, one reason might be that the journals
do not support 3D PDFs. However, a further—and probably
more serious reason—might be that most clinicians andscientists do not know about this feature. Although thishypothesis could not be investigated with our systematic review
and it should be investigated in future research, the pure

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Potential of Three-Dimensional Portable Document
Format and Future Research
Although it is not widely disseminated yet, the first approaches
are very promising and show that 3D PDF has a high potential
in clinical communication, in biomedical research, and inresearch in general. 3D PDF offers the possibility to simply
distribute atlases of, for example, human or animal anatomy asdemonstrated in the 3D Atlas of Human Embryology[110,200,230] or in the Visible Korean anatomy database[62,231]. It is, therefore, an alternative to traditional 2D atlases
in which much of information gets lost because of the projectionof a 3D object to a 2D plane [18]. Furthermore, 3D PDFs caneasily be distributed via the internet or shared via emails, making
it unnecessary to travel to museums all around the world, forexample.
Future research should mainly focus on the development ofsoftware tools that can be used easily by everyone and withoutcharge. Furthermore, it should be investigated (eg, by meansof surveys) how well-known 3D PDF is in the biomedical andclinical community. Besides, simple instruction manuals forthe existing tools are needed and should be disseminated further.
Additionally, the routine usage of 3D PDF in clinicalapplications should be further promoted and evaluated for otherdisciplines than liver surgery planning. Finally, future research
could also consider other domains than the biomedical one.

Limitations
Our review is subject to some limitations. First, an unknown
number of articles that would fulfill the inclusion criteria may
not have been found. The most probable reason for missing
such an article might be that authors developed the 3D PDF on
their own (without any of the tool articles) and that theavailability of a 3D PDF was not necessarily pointed out in the
JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.108(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

abstract or in the available full text. Another reason might bethat a tool article was used, but not properly cited. However,
200 articles can be considered a solid basis for a review.

Third, a systematic review is not a suitable method to investigate
the personal reasons for researchers not using 3D PDF. Other
methods such as surveys are needed for this.

Based on our expertise and the high relevance of 3D PDFs in
the investigated domain, the scope of this review was
intentionally limited to biomedical publications. Other scientificdisciplines are known to use 3D PDF as well (eg, astronomy[232], paleontology [233,234], and chemistry [235]) and may
yield different results.

Conclusions

Second, as already mentioned, many of the protocols presented
previously are based on commercial software that is no longer
available and that has no successor providing the samefunctionality. To cover all available articles, they are stillpresented, but the use of the more modern protocols is
recommended.

The use of 3D PDF for visualization purposes in real medical
use cases or in biomedical publications is not yet fully accepted,
although the necessary technique is available and there are manyarguments in favor of 3D PDF. In this review paper, a wide
range of examples for applied 3D PDF technology and a variety
of protocols and software tools for the creation of relevant
documents and files are presented. It aims to draw attention tothis valuable technology, to demonstrate the possibilities, tohelp interested readers find a suitable solution, and lower thethreshold for its use.

Acknowledgments
We acknowledge support by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) and Friedrich-Alexander
University Erlangen-Nürnberg within the funding program Open Access Publishing. Linda Becker was supported by the Bavarian
Equal Opportunities Sponsorship—Förderung von Frauen in Forschung und Lehre (FFL)—Promoting Equal Opportunities for
Women in Research and Teaching.

Authors' Contributions
AN and LB conceived the study. AN performed the literature search. AN and LB screened the search results. AN and LB analyzed
the search results. AN and LB drafted and wrote the manuscript.

Conflicts of Interest
None declared.

Multimedia Appendix 1
3D PDF version of Figure 1.
[PDF File (Adobe PDF File), 3MB - medinform_v6i3e10295_app1.pdf ]Multimedia Appendix 2
3D PDF version of this article (best viewed with Adobe Reader 9 or later).
[PDF File (Adobe PDF File), 3MB - medinform_v6i3e10295_app2.pdf ]Multimedia Appendix 3
Overview over the search strategy and all articles that were found and processed.
[XLSX File (Microsoft Excel File), 143KB - medinform_v6i3e10295_app3.xlsx ]Multimedia Appendix 4
Detailed analysis of all eligible articles.
[XLSX File (Microsoft Excel File), 68KB - medinform_v6i3e10295_app4.xlsx ]Multimedia Appendix 5
Detailed analysis of articles that present a protocol or a software for 3D PDF creation.
[XLSX File (Microsoft Excel File), 16KB - medinform_v6i3e10295_app5.xlsx ]References
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.109(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
1.

2.

3.

4.

5.
6.

7.

8.

9.

10.

11.

12.
13.

14.

15.

16.
17.

18.
19.
20.

21.
22.

International Organization for Standardization. 2017. ISO 32000-2:2017 Document management--portable document
format--Part 2: PDF 2.0 URL: https://www.iso.org/standard/63534.html [accessed 2018-03-01] [WebCite Cache ID
6xb077p9M]
Rico RD, Juanes Méndez JA, Mavar-Haramija M, Reina Perticone MA, Prats-Galino A. Possibilities of application of
3D-PDF documents to represent models and tridimensional images in medicine. 2014 Presented at: Second International
Conference on Technological Ecosystems for Enhancing Multiculturality; Oct 1-3, 2014; Salamanca, Spain p. 53-56. [doi:10.1145/2669711.2669878]
Newe A, Becker L, Schenk A. Application and evaluation of interactive 3D PDF for presenting and sharing planning results
for liver surgery in clinical routine. PLoS One 2014;9(12):e115697 [FREE Full text] [doi: 10.1371/journal.pone.0115697][Medline: 25551375]
Mavar-Haramija M, Prats-Galino A, Méndez JA, Puigdelívoll-Sánchez A, de Notaris M. Interactive 3D-PDF presentations
for the simulation and quantification of extended endoscopic endonasal surgical approaches. J Med Syst 2015 Oct;39(10):127.
[doi: 10.1007/s10916-015-0282-7] [Medline: 26306875]
ASTM International. 2008 Mar 01. PDF Healthcare best practices guide published URL: https://www.astm.org/newsroom/
pdf-healthcare-best-practices-guide-published [accessed 2018-03-01] [WebCite Cache ID 6xazlrn0L]
Association for Information and Image Management. 2008. Best practices-implementation guide for the portable document
format healthcare URL: http://www.aiim.org/documents/standards/PDF-h_Implementation_Guide_2008.pdf [accessed
2018-05-01] [WebCite Cache ID 6QkHzfRns]
Adobe Systems Incorporated. 2008 Jul 01. Document management—Portable document format—Part 1: PDF 1.7 URL:
http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/pdf/pdfs/PDF32000_2008.pdf [accessed 2018-05-01] [WebCite
Cache ID 6S8xkrvPw]
International Organization for Standardization. 2013. ISO 32000-1:2008 Document management--portable document
format--Part 1: PDF 1.7 URL: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=51502[accessed 2018-05-01] [WebCite Cache ID 6QkFZgVk9]
PDF Association. PDF 2.0: The worldwide standard for electronic documents has evolved URL: https://www.pdfa.org/
intro/pdf-2-0-the-worldwide-standard-for-electronic-documents-has-evolved/ [accessed 2018-05-01] [WebCite Cache ID
6xb0LmX8M]
Apple. 2004 May 27. MacOS X: System overview: the graphics and windowing environment URL: https://web.archive.org/
web/20040925095929/http://developer.apple.com/documentation/MacOSX/Conceptual/SystemOverview/SystemArchitecture/
chapter_3_section_4.html [accessed 2018-05-01] [WebCite Cache ID 707QBVPLP]
Adobe Systems Incorporated. PDF Reference, Fifth Edition, Adobe Portable Document Format, Version 1.6. 2004. URL:
https://wwwimages2.adobe.com/content/dam/acom/en/devnet/pdf/pdf_reference_archive/PDFReference16.pdf [accessed
2018-05-01] [WebCite Cache ID 6xb0tF2Lx]
3dtest. 2004 Mar 11. Q&A: Richard D Benoit, Intel URL: http://www.3d-test.com/contenu.php?id=418 [accessed 2018-05-01][WebCite Cache ID 6xb18rG94]
ECMA International. Standard ECMA-363, Universal 3D File Format, 4th edition. Geneva: Ecma International; 2007 Jun.
URL: http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-363%204th%20Edition.pdf [accessed
2018-05-01] [WebCite Cache ID 6QkI44nqn]
Dakan M. Acrobat Insider for AEC Professionals. 2006 Apr. Enter the third dimension: Acrobat 7.0 Professional makes
working with 3D content easy and convenient URL: http://www.cadalyst.com/aec/acrobat-insider-9-pdf-4614 [accessed
2018-06-12] [WebCite Cache ID 707QUWbUJ]
Adobe Systems Incorporated. Adobe Supplement to the ISO 32000, BaseVersion: 1.7, ExtensionLevel: 3. 2008 Jun. URL:
http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/pdf/pdfs/adobe_supplement_iso32000.pdf [accessed 2018-05-01][WebCite Cache ID 6xaz93gNe]
Vora K. Adobe Conversations. 2010 May 24. 3D Solutions update URL: https://blogs.adobe.com/conversations/2010/05/
3d_solutions_update.html [accessed 2018-03-01] [WebCite Cache ID 6xaz2a98d]
Adobe Systems Incorporated. PDF Reference, Second edition, Adobe Portable Document Format, Version 1.3. URL: https://wwwimages2.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/pdf_reference_archives/PDFReference13.pdf [accessed
2018-05-01] [WebCite Cache ID 6xb0wzaki]
Tory M, Möller T. Human factors in visualization research. IEEE Trans Vis Comput Graph 2004;10(1):72-84. [doi:10.1109/TVCG.2004.1260759] [Medline: 15382699]
Murienne J, Ziegler A, Ruthensteiner B. A 3D revolution in communicating science. Nature 2008 May 22;453(7194):450.
[doi: 10.1038/453450d] [Medline: 18497796]
Barnes DG, Vidiassov M, Ruthensteiner B, Fluke CJ, Quayle MR, McHenry CR. Embedding and publishing interactive,
3-dimensional, scientific figures in Portable Document Format (PDF) files. PLoS One 2013;8(9):e69446 [FREE Full text][doi: 10.1371/journal.pone.0069446] [Medline: 24086243]
Maunsell J. Announcement regarding supplemental material. J Neurosci 2010;30(32):10599-10600 [FREE Full text]
Elsevier. Interactive U3D models URL: https://www.elsevier.com/authors/author-services/enrichments/interactive-u3d-models[accessed 2018-03-01] [WebCite Cache ID 6xazzYWWN]

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.110(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
23.
24.
25.
26.

27.
28.
29.

30.
31.
32.
33.
34.
35.
36.
37.
38.

39.
40.

41.

42.
43.
44.
45.

46.
47.
48.
49.

Aalbersberg IJ, Cos Alvarez P, Jomier J, Marion C, Zudilova-Seinstra E. Bringing 3D visualization into the online research
article. ISU 2014 Sep 19;34(1-2):27-37. [doi: 10.3233/ISU-140721]
NCBI. PubMed URL: https://www.ncbi.nlm.nih.gov/pubmed [accessed 2018-06-12] [WebCite Cache ID 707iIG5La]
Google Scholar. URL: https://scholar.google.de/ [accessed 2018-06-12] [WebCite Cache ID 707iKPOlo]
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: thePRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [doi: 10.1371/journal.pmed.1000097] [Medline:
19621072]
Tabernero RR, Juanes MJ, Prats GA. New generation of three-dimensional tools to learn anatomy. J Med Syst 2017
May;41(5):88. [doi: 10.1007/s10916-017-0725-4] [Medline: 28405946]
Handschuh S, Schwaha T, Metscher BD. Showing their true colors: a practical approach to volume rendering from serial
sections. BMC Dev Biol 2010 Apr 21;10:41 [FREE Full text] [doi: 10.1186/1471-213X-10-41] [Medline: 20409315]
Prilusky J, Hodis E, Canner D, Decatur WA, Oberholser K, Martz E, et al. Proteopedia: a status report on the collaborative,
3D web-encyclopedia of proteins and other biomolecules. J Struct Biol 2011 Aug;175(2):244-252. [doi:10.1016/j.jsb.2011.04.011]
Ziegler A. Broad application of non-invasive imaging techniques to echinoids and other echinoderm taxa. Zoosymposia
2012;7(1):53-70. [doi: 10.11646/zoosymposia.7.1.6]
Webb HK, Boshkovikj V, Fluke CJ, Truong VK, Hasan J, Baulin VA, et al. Bacterial attachment on sub-nanometricallysmooth titanium substrata. Biofouling 2013;29(2):163-170. [doi: 10.1080/08927014.2012.757697] [Medline: 23327438]
Margulies DS, Böttger J, Watanabe A, Gorgolewski KJ. Visualizing the human connectome. Neuroimage 2013 Oct
15;80:445-461 [FREE Full text] [doi: 10.1016/j.neuroimage.2013.04.111] [Medline: 23660027]
Breure L, Hoogerwerf M, van Horik R. Xpos’re: a tool for rich Internet publications. Digit Hum Quart 2014;8(2).
Malkowsky Y, Jochum A. Three-dimensional reconstructions of pallial eyes in Pectinidae (Mollusca: Bivalvia). Acta Zool
2014 Jan 31;96(2):167-173. [doi: 10.1111/azo.12064]
Carbayo F, Francoy TM, Giribet G. Non-destructive imaging to describe a new species of land planarian (Platyhelminthes,
Tricladida). Zool Scr 2016 Mar 03;45(5):566-578. [doi: 10.1111/zsc.12175]
Chung BS, Kim J, Chung MS. Integrated comics and visible Korean movies for laypeople's learning of systemic anatomy.
Int J Morphol 2017 Sep;35(3):883-887. [doi: 10.4067/S0717-95022017000300014]
Laaß M, Schillinger B, Werneburg I. Neutron tomography and X-ray tomography as tools for the morphological investigationof non-mammalian Synapsids. Physics Procedia 2017;88:100-108. [doi: 10.1016/j.phpro.2017.06.013]
Schachner ER, Sedlmayr JC, Schott R, Lyson TR, Sanders RK, Lambertz M. Pulmonary anatomy and a case of unilateralaplasia in a common snapping turtle (Chelydra serpentina): developmental perspectives on cryptodiran lungs. J Anat 2017
Dec;231(6):835-848. [doi: 10.1111/joa.12722] [Medline: 29063595]
Chung BS, Chung MS. Homepage to distribute the anatomy learning contents including Visible Korean products, comics,
and books. Anat Cell Biol 2018 Mar;51(1):7-13 [FREE Full text] [doi: 10.5115/acb.2018.51.1.7] [Medline: 29644104]
Ziegler A, Mietchen D, Faber C, von Hausen W, Schöbel C, Sellerer M, et al. Effectively incorporating selected multimedia
content into medical publications. BMC Med 2011 Feb 17;9:17 [FREE Full text] [doi: 10.1186/1741-7015-9-17] [Medline:
21329532]
Mavar-Haramija M, Prats-Galino A, Méndez J, Puigdelívoll-Sánchez A, de Notaris M. Simulation of surgical procedures
and associated quantification methods by 3D PDF documents. 2014 Presented at: Second International Conference on
Technological Ecosystems for Enhancing Multiculturality; Oct 1-3, 2014; Salamanca, Spain p. 57-62. [doi:10.1145/2669711.2669879]
Park JS, Chung BS, Chung MS. Digital anatomy using the surface models in portable document format file for self-learningand evaluation. Digit Med 2017;3(3):133. [doi: 10.4103/digm.digm_29_17]
Wirkner CS, Richter S. Evolutionary morphology of the circulatory system in Peracarida (Malacostraca; Crustacea).
Cladistics 2009;26(2):143-167. [doi: 10.1111/j.1096-0031.2009.00278.x]
Ziegler A, Kunth M, Mueller S, Bock C, Pohmann R, Schröder L, et al. Application of magnetic resonance imaging inzoology. Zoomorphology 2011 Oct 13;130(4):227-254. [doi: 10.1007/s00435-011-0138-8]
Ziegler A, Stock SR, Menze BH, Smith AB. Macro- and microstructural diversity of sea urchin teeth revealed by large-scale
mircro-computed tomography survey. In: Proc SPIE 8506: Developments in X-Ray Tomography VIII. 2012 Presented at:
SPIE Optical Engineering + Applications; Oct 17, 2012; San Diego, CA. [doi: 10.1117/12.930832]
Gaweł D, Danielewicz K, Nowak M. The method of geometrical comparison of 3-dimensional objects created from DICOM
images. Stud Health Technol Inform 2012;176:273-276. [Medline: 22744507]
van de Kamp T, Ershov A, Dos Santos Rolo T, Riedel A, Baumbach T. Insect imaging at the ANKA synchrotron radiation
facility. Entomologie heute 2013;25:147-160.
Lautenschlager S, Rücklin M. Beyond the print-virtual paleontology in science publishing, outreach, and education. J
Paleontol 2014 Jul 01;88(4):727-734 [FREE Full text] [doi: 10.1666/13-085] [Medline: 26306051]
Jocks I, Livingstone D, Rea PM. An investigation to examine the most appropriate methodology to capture historical andmodern preserved anatomical specimens for use in the digital age to improve access: a pilot study. In: Proceedings of

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.111(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

50.
51.
52.
53.
54.

55.
56.
57.
58.
59.
60.

61.

62.

63.
64.
65.

66.

67.

68.
69.
70.

71.

72.

INTED2015 Conference. 2015 Presented at: 9th International Technology, Education and Development Conference; Mar
2-4, 2015; Madrid, Spain p. 6377-6386.
Xavier JC, Allcock AL, Cherel Y, Lipinski MR, Pierce GJ, Rodhouse PG, et al. Future challenges in cephalopod research.
J Mar Biol Assoc 2014 Jun 26;95(05):999-1015. [doi: 10.1017/S0025315414000782]
Newe A, Becker L. Using interactive 3D PDF for exploring complex biomedical data: experiences and solutions. Stud
Health Technol Inform 2016;228:740-744. [doi: 10.3233/978-1-61499-678-1-740] [Medline: 27577484]
Williston AD. Charles Minot and the Harvard Embryological Collection: over a century of development. Breviora 2016
Apr;547(1):1-13. [doi: 10.3099/0006-9698-547.00.1]
Lebrun R, Orliac MJ. Morphomuseum: an online platform for publication and storage of virtual specimens. Paleontol Soc
Pap 2017 Apr 27;22:183-195. [doi: 10.1017/scs.2017.14]
Elbashti ME, Aswehlee AM, Sumita YI, Hattori M, Taniguchi H. The role of portable documentation format inthree-dimensional interactive visualization in maxillofacial prosthetics. Int J Prosthodont 2018 Mar 28. [doi:10.11607/ijp.5475] [Medline: 29590664]
Kumar P, Ziegler A, Ziegler J, Uchanska-Ziegler B, Ziegler A. Grasping molecular structures through publication-integrated
3D models. Trends Biochem Sci 2008 Sep;33(9):408-412. [doi: 10.1016/j.tibs.2008.06.004] [Medline: 18672371]
Barnes DG, Fluke CJ. Incorporating interactive three-dimensional graphics in astronomy research papers. New Astron2008 Nov;13(8):599-605. [doi: 10.1016/j.newast.2008.03.008]
Ruthensteiner B, Hess M. Embedding 3D models of biological specimens in PDF publications. Microsc Res Tech 2008
Nov;71(11):778-786. [doi: 10.1002/jemt.20618] [Medline: 18785246]
Kumar P, Ziegler A, Grahn A, Hee CS, Ziegler A. Leaving the structural ivory tower, assisted by interactive 3D PDF.
Trends Biochem Sci 2010 Aug;35(8):419-422. [doi: 10.1016/j.tibs.2010.03.008] [Medline: 20541422]
Ruthensteiner B, Baeumler N, Barnes DG. Interactive 3D volume rendering in biomedical publications. Micron 2010
Oct;41(7):886.e1-886.e17. [doi: 10.1016/j.micron.2010.03.010] [Medline: 20562000]
de Boer BA, Soufan AT, Hagoort J, Mohun TJ, van den Hoff MJ, Hasman A, et al. The interactive presentation of 3D
information obtained from reconstructed datasets and 3D placement of single histological sections with the 3D portable
document format. Development 2011 Jan;138(1):159-167 [FREE Full text] [doi: 10.1242/dev.051086] [Medline: 21138978]
Danz JC, Katsaros C. Three-dimensional portable document format: a simple way to present 3-dimensional data in anelectronic publication. Am J Orthod Dentofacial Orthop 2011 Aug;140(2):274-276. [doi: 10.1016/j.ajodo.2011.04.010][Medline: 21803267]
Shin DS, Chung MS, Park JS, Park HS, Lee S, Moon YL, et al. Portable document format file showing the surface models
of cadaver whole body. J Korean Med Sci 2012 Aug;27(8):849-856 [FREE Full text] [doi: 10.3346/jkms.2012.27.8.849][Medline: 22876049]
Phelps A, Naeger DM, Marcovici P. Embedding 3D radiology models in portable document format. AJR Am J Roentgenol
2012 Dec;199(6):1342-1344. [doi: 10.2214/AJR.12.8716] [Medline: 23169728]
Lautenschlager S. Palaeontology in the third dimension: a comprehensive guide for the integration of three-dimensional
content in publications. Paläontol Z 2013 May 31;88(1):111-121. [doi: 10.1007/s12542-013-0184-2]
Mavar-Haramija M, Prats-Galino A, Escuder CB, Juanes Méndez JA, Puigdelívoll-Sánchez A. 3D PDF Technology
combined with JavaScript functions enables the creation and visualization of interactive 3D presentations. 2013 Presented
at: First International Conference on Technological Ecosystem for Enhancing Multiculturality; Nov 14-15, 2013; Salamanca,
Spain p. 67-72. [doi: 10.1145/2536536.2536548]
Newe A, Ganslandt T. Simplified generation of biomedical 3D surface model data for embedding into 3D portable document
format (PDF) files for publication and education. PLoS One 2013;8(11):e79004 [FREE Full text] [doi:10.1371/journal.pone.0079004] [Medline: 24260144]
van de Kamp T, dos Santos Rolo T, Vagovič P, Baumbach T, Riedel A. Three-dimensional reconstructions come tolife--interactive 3D PDF animations in functional morphology. PLoS One 2014;9(7):e102355 [FREE Full text] [doi:10.1371/journal.pone.0102355] [Medline: 25029366]Newe A. Towards an easier creation of three-dimensional data for embedding into scholarly 3D PDF (Portable Document
Format) files. PeerJ 2015;3:e794 [FREE Full text] [doi: 10.7717/peerj.794] [Medline: 25780759]
Newe A. Enriching scientific publications with interactive 3D PDF: an integrated toolbox for creating ready-to-publish
figures. PeerJ Comp Sci 2016;2:e64. [doi: 10.7717/peerj-cs.64]
Zhang N, Li Q, Jia H, Zhang M. U3D File format analyzing and 3DPDF generating method. In: Wang Y, editor. Advancesin Image and Graphics Technologies. IGTA 2017. Communications in Computer and Information Science, vol 757.
Singapore: Springer; 2017:136-146.
Brandner J, Newe A, Aichinger W, Linda B. An open source tool for creating model files for virtual volume rendering in
PDF documents. In: Maier A, Deserno T, Handels H, Maier-Hein K, Palm C, Tolxdorff T, editors. Bildverarbeitung fürdie Medizin 2018. Informatik aktuell. Berlin: Springer; 2018:133-138.
Ganesan A, Wang F, Falzon C. Intramolecular interactions of L-phenylalanine: valence ionization spectra and orbital
momentum distributions of its fragment molecules. J Comput Chem 2011 Feb;32(3):525-535. [doi: 10.1002/jcc.21639][Medline: 20806261]

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.112(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
73.

74.

75.

76.

77.

78.
79.

80.
81.
82.
83.

84.

85.
86.

87.

88.
89.
90.
91.

92.
93.

94.

95.

Truong VK, Lapovok R, Estrin YS, Rundell S, Wang JY, Fluke CJ, et al. The influence of nano-scale surface roughness
on bacterial adhesion to ultrafine-grained titanium. Biomaterials 2010 May;31(13):3674-3683. [doi:10.1016/j.biomaterials.2010.01.071] [Medline: 20163851]
Strickfaden H, Zunhammer A, van Koningsbruggen S, Köhler D, Cremer T. 4D chromatin dynamics in cycling cells:
Theodor Boveri's hypotheses revisited. Nucleus 2010;1(3):284-297 [FREE Full text] [doi: 10.4161/nucl.1.3.11969] [Medline:
21327076]
Loll B, Rückert C, Hee CS, Saenger W, Uchanska-Ziegler B, Ziegler A. Loss of recognition by cross-reactive T cells andits relation to a C-terminus-induced conformational reorientation of an HLA-B*2705-bound peptide. Protein Sci 2011
Feb;20(2):278-290 [FREE Full text] [doi: 10.1002/pro.559] [Medline: 21280120]
Lintner NG, Frankel KA, Tsutakawa SE, Alsbury DL, Copié V, Young MJ, et al. The structure of the CRISPR-associatedprotein Csa3 provides insight into the regulation of the CRISPR/Cas system. J Mol Biol 2011 Jan 28;405(4):939-955 [FREEFull text] [doi: 10.1016/j.jmb.2010.11.019] [Medline: 21093452]
Uchanska-Ziegler B, Loll B, Fabian H, Hee CS, Saenger W, Ziegler A. HLA class I-associated diseases with a suspectedautoimmune etiology: HLA-B27 subtypes as a model system. Eur J Cell Biol 2012 Apr;91(4):274-286. [doi:10.1016/j.ejcb.2011.03.003] [Medline: 21665321]
Hee CS, Fabian H, Uchanska-Ziegler B, Ziegler A, Loll B. Comparative biophysical characterization of chicken
β2-microglobulin. Biophys Chem 2012 Jun;167:26-35. [doi: 10.1016/j.bpc.2012.04.001] [Medline: 22695053]
Balke M, Ruthensteiner B, Warikar EL, Neven K, Hendrich L. Two new species of Limbodessus diving beetles from New
Guinea - short verbal descriptions flanked by online content (digital photography, μCT scans, drawings and DNA sequencedata). Biodivers Data J 2015(3):e7096 [FREE Full text] [doi: 10.3897/BDJ.3.e7096] [Medline: 26752969]
Chung BS, Kwon K, Shin BS, Chung MS. Peeled and piled volume models of the stomach made from a cadaver's sectioned
images. Int J Morphol 2016 Sep;34(3):939-944. [doi: 10.4067/S0717-95022016000300020]
Shin DS, Park SK. Surface reconstruction and optimization of cerebral cortex for application use. J Craniofac Surg 2016
Mar;27(2):489-492. [doi: 10.1097/SCS.0000000000002352] [Medline: 26854785]
Baeumler N, Haszprunar G, Ruthensteiner B. 3D interactive microanatomy of Omalogyra atomus (Philippi, 1841)(Gastropoda, Heterobranchia, Omalogyridae). Zoosymposia 2008;1:101-118. [doi: 10.11646/zoosymposia.1.1.9]
Ziegler A, Faber C, Mueller S, Bartolomaeus T. Systematic comparison and reconstruction of sea urchin (Echinoidea)internal anatomy: a novel approach using magnetic resonance imaging. BMC Biol 2008 Jul 23;6:33 [FREE Full text] [doi:10.1186/1741-7007-6-33] [Medline: 18651948]
Kumar P, Vahedi-Faridi A, Saenger W, Ziegler A, Uchanska-Ziegler B. Conformational changes within the
HLA-A1:MAGE-A1 complex induced by binding of a recombinant antibody fragment with TCR-like specificity. Protein
Sci 2009 Jan;18(1):37-49 [FREE Full text] [doi: 10.1002/pro.4] [Medline: 19177349]
Selvam L, Vasilyev V, Wang F. Methylation of zebularine: a quantum mechanical study incorporating interactive 3D pdf
graphs. J Phys Chem B 2009 Aug 20;113(33):11496-11504. [doi: 10.1021/jp901678g] [Medline: 19637931]
Neusser TP, Heß M, Schrödl M. Tiny but complex - interactive 3D visualization of the interstitial acochlidian gastropod
Pseudunela cornuta (Challis, 1970). Front Zool 2009 Sep 11;6:20 [FREE Full text] [doi: 10.1186/1742-9994-6-20] [Medline:
19747373]
Kumar P, Vahedi-Faridi A, Saenger W, Merino E, López de Castro JA, Uchanska-Ziegler B, et al. Structural basis for T
cell alloreactivity among three HLA-B14 and HLA-B27 antigens. J Biol Chem 2009 Oct 23;284(43):29784-29797 [FREEFull text] [doi: 10.1074/jbc.M109.038497] [Medline: 19617632]
Ivanova EP, Truong VK, Wang JY, Berndt CC, Jones RT, Yusuf II, et al. Impact of nanoscale roughness of titanium thinfilm surfaces on bacterial retention. Langmuir 2010 Feb 02;26(3):1973-1982. [doi: 10.1021/la902623c] [Medline: 19842625]
Mitik-Dineva N, Wang J, Truong VK, Stoddart PR, Alexander MR, Albutt DJ, et al. Bacterial attachment on optical fibre
surfaces. Biofouling 2010 May;26(4):461-471. [doi: 10.1080/08927011003753399] [Medline: 20358429]
Ziegler A, Ogurreck M, Steinke T, Beckmann F, Prohaska S, Ziegler A. Opportunities and challenges for digital morphology.
Biol Direct 2010 Jul 06;5:45 [FREE Full text] [doi: 10.1186/1745-6150-5-45] [Medline: 20604956]
Ziegler A, Mooi R, Rolet G, De Ridder C. Origin and evolutionary plasticity of the gastric caecum in sea urchins(Echinodermata: Echinoidea). BMC Evol Biol 2010 Oct 18;10:313 [FREE Full text] [doi: 10.1186/1471-2148-10-313][Medline: 20955602]
Ruthensteiner B, Schropel V, Haszprunar G. Anatomy and affinities of Micropilina minuta Waren, 1989 (Monoplacophora:
Micropilinidae). J Mollus Stud 2010 Jun 09;76(4):323-332. [doi: 10.1093/mollus/eyq013]
Hartmann H, Hess M, Haszprunar G. Interactive 3D anatomy and affinities of Bathysciadiidae (Gastropoda, Cocculinoidea):
Deep-sea limpets feeding on decaying cephalopod beaks. J Morphol 2011 Mar;272(3):259-279. [doi: 10.1002/jmor.10910][Medline: 21312226]
Brenzinger B, Neusser TP, Jörger KM, Schrödl M. Integrating 3D microanatomy and molecules: natural history of the
Pacific freshwater slug Strubellia Odhner, 1937 (Heterobranchia: Acochlidia), with description of a new species. J Mollus
Stud 2011;77(4):351-374. [doi: 10.1093/mollus/eyr027]
Klußmann-Fricke B, Prendini L, Wirkner CS. Evolutionary morphology of the hemolymph vascular system in scorpions:a character analysis. Arthropod Struct Dev 2012 Nov;41(6):545-560. [doi: 10.1016/j.asd.2012.06.002] [Medline: 22735399]

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

Newe & Becker

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.113(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

96.

Lipke E, Michalik P. Formation of primary sperm conjugates in a haplogyne spider (Caponiidae, Araneae) with remarkson the evolution of sperm conjugation in spiders. Arthropod Struct Dev 2012 Nov;41(6):561-573. [doi:10.1016/j.asd.2012.08.001] [Medline: 22982877]
97. Genolini C, Pingault JB, Driss T, Côté S, Tremblay RE, Vitaro F, et al. KmL3D: a non-parametric algorithm for clustering
joint trajectories. Comput Methods Programs Biomed 2013 Jan;109(1):104-111. [doi: 10.1016/j.cmpb.2012.08.016] [Medline:
23127283]
98. Faulwetter S, Vasileiadou A, Kouratoras M, Thanos D, Arvanitidis C. Micro-computed tomography:
Introducing new dimensions to taxonomy. Zookeys 2013;4(263):1-45 [FREE Full text] [doi: 10.3897/zookeys.263.4261][Medline: 23653515]
99. Düring DN, Ziegler A, Thompson CK, Ziegler A, Faber C, Müller J, et al. The songbird syrinx morphome: athree-dimensional, high-resolution, interactive morphological map of the zebra finch vocal organ. BMC Biol 2013 Jan
08;11:1 [FREE Full text] [doi: 10.1186/1741-7007-11-1] [Medline: 23294804]
100. Hawe A, Heß M, Haszprunar G. 3D reconstruction of the anatomy of the ovoviviparous (?) freshwater gastropod Borysthenia
naticina (Menke, 1845)(Ectobranchia: Valvatidae). J Mollus Stud 2013;79(3):191-201. [doi: 10.1093/mollus/eyt018]
101. Kato A, Ziegler A, Higuchi N, Nakata K, Nakamura H, Ohno N. Aetiology, incidence and morphology of the C-shaped
root canal system and its impact on clinical endodontics. Int Endod J 2014 Nov;47(11):1012-1033 [FREE Full text] [doi:10.1111/iej.12256] [Medline: 24483229]
102. Haszprunar G, Graf L, Hess M. 3D-anatomy of the ctenoglossate limpet Kaiparapelta (Vetigastropoda: Lepetelloidea). J
Mollus Stud 2014 Jan 20;80(1):84-98. [doi: 10.1093/mollus/eyt051]
103. Hawe A, Paroll C, Haszprunar G. Interactive 3D-anatomical reconstruction and affinities of the hot-vent gastropod
Xylodiscula analoga Waren & Bouchet, 2001 (Ectobranchia). J Mollus Stud 2014 Apr 08;80(3):315-325. [doi:10.1093/mollus/eyu017]
104. Judge J, Haszprunar G. The anatomy of Lepetella sierrai (Vetigastropoda, Lepetelloidea): implications for reproduction,
feeding, and symbiosis in lepetellid limpets. Invertebr Biol 2014 Jul 02;133(4):324-339. [doi: 10.1111/ivb.12064]
105. Ziegler A. Rediscovery of an internal organ in heart urchins (Echinoidea: Spatangoida): morphology and evolution of theintestinal caecum. Org Divers Evol 2014 Jul 9;14(4):383-395. [doi: 10.1007/s13127-014-0178-2]
106. Ziegler A, Lenihan J, Zachos LG, Faber C, Mooi R. Comparative morphology and phylogenetic significance of Gregory’s
diverticulum in sand dollars (Echinoidea: Clypeasteroida). Org Divers Evol 2015 Aug 28;16(1):141-166. [doi:10.1007/s13127-015-0231-9]
107. Kato A, Ziegler A, Utsumi M, Ohno K, Takeichi T. Three-dimensional imaging of internal tooth structures: applications
in dental education. J Oral Biosci 2016 Aug;58(3):100-111. [doi: 10.1016/j.job.2016.05.004]
108. Kunze T, Heß M, Haszprunar G. 3D-interactive microanatomy of Ventsia tricarinata Warén & Bouchet, 1993 (Vetigastropoda:
Seguenzioidea) from Pacific hydrothermal vents. J Mollus Stud 2016 Feb 17;82(3):366-377. [doi: 10.1093/mollus/eyw002]
109. Day NJ, Earnshaw D, Salazar-Ferrer P, Walsh CJ. Preoperative mapping of fistula-in-ano: a new three-dimensional
MRI-based modelling technique. Colorectal Dis 2013 Nov;15(11):e699-e701. [doi: 10.1111/codi.12438] [Medline: 24119050]
110. de Bakker BS, de Jong KH, Hagoort J, Oostra RJ, Moorman AF. Towards a 3-dimensional atlas of the developing humanembryo: the Amsterdam experience. Reprod Toxicol 2012 Sep;34(2):225-236. [doi: 10.1016/j.reprotox.2012.05.087][Medline: 22640940]
111. de Doer BA, van den Berg G, de Boer PAJ, Moorman AFM, Ruijter JM. Growth of the developing mouse heart: an interactive
qualitative and quantitative 3D atlas. Dev Biol 2012 Aug 15;368(2):203-213 [FREE Full text] [doi:10.1016/j.ydbio.2012.05.001] [Medline: 22617458]
112. Holliday CM, Tsai HP, Skiljan RJ, George ID, Pathan S. A 3D interactive model and atlas of the jaw musculature of
Alligator mississippiensis. PLoS One 2013;8(6):e62806 [FREE Full text] [doi: 10.1371/journal.pone.0062806] [Medline:
23762228]
113. Lautenschlager S, Bright JA, Rayfield EJ. Digital dissection - using contrast-enhanced computed tomography scanning toelucidate hard- and soft-tissue anatomy in the Common Buzzard Buteo buteo. J Anat 2014 Apr;224(4):412-431 [FREEFull text] [doi: 10.1111/joa.12153] [Medline: 24350638]
114. Quayle MR, Barnes DG, Kaluza OL, McHenry CR. An interactive three dimensional approach to anatomical description-the
jaw musculature of the Australian laughing kookaburra (Dacelo novaeguineae). PeerJ 2014;2:e355 [FREE Full text] [doi:10.7717/peerj.355] [Medline: 24860694]
115. Jensen B, Moorman AF, Wang T. Structure and function of the hearts of lizards and snakes. Biol Rev Camb Philos Soc
2014 May;89(2):302-336. [doi: 10.1111/brv.12056] [Medline: 23998743]
116. Scherz MD, Ruthensteiner B, Vences M, Glaw F. A new microhylid frog, genus Rhombophryne, from northeastern
Madagascar, and a re-description of R. serratopalpebrosa using micro-computed tomography. Zootaxa 2014 Sep
10;3860(6):547-560. [doi: 10.11646/zootaxa.3860.6.3] [Medline: 25283290]
117. Xu XG. An exponential growth of computational phantom research in radiation protection, imaging, and radiotherapy: areview of the fifty-year history. Phys Med Biol 2014 Sep 21;59(18):R233-R302 [FREE Full text] [doi:10.1088/0031-9155/59/18/R233] [Medline: 25144730]

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.114(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

118. Wu Y, Dabhoiwala NF, Hagoort J, Shan JL, Tan LW, Fang BJ, et al. 3D Topography of the young adult anal sphincter
complex reconstructed from undeformed serial anatomical sections. PLoS One 2015;10(8):e0132226 [FREE Full text] [doi:10.1371/journal.pone.0132226] [Medline: 26305117]
119. Prötzel D, Ruthensteiner B, Scherz MD, Glaw F. Systematic revision of the Malagasy chameleons Calumma boettgeri andC. linotum (Squamata: Chamaeleonidae). Zootaxa 2015 Nov 25;4048(2):211-231. [Medline: 26624746]
120. Porro LB, Richards CT. Digital dissection of the model organism Xenopus laevis using contrast-enhanced computed
tomography. J Anat 2017 Aug;231(2):169-191. [doi: 10.1111/joa.12625] [Medline: 28547827]
121. Truong VK, Rundell S, Lapovok R, Estrin Y, Wang JY, Berndt CC, et al. Effect of ultrafine-grained titanium surfaces onadhesion of bacteria. Appl Microbiol Biotechnol 2009 Jul;83(5):925-937. [doi: 10.1007/s00253-009-1944-5] [Medline:
19296098]
122. Bangs F, Welten M, Davey MG, Fisher M, Yin Y, Downie H, et al. Identification of genes downstream of the Shh signallingin the developing chick wing and syn-expressed with Hoxd13 using microarray and 3D computational analysis. Mech Dev
2010;127(9-12):428-441 [FREE Full text] [doi: 10.1016/j.mod.2010.08.001] [Medline: 20708683]
123. Hee CS, Gao S, Loll B, Miller MM, Uchanska-Ziegler B, Daumke O, et al. Structure of a classical MHC class I moleculethat binds “non-classical” ligands. PLoS Biol 2010 Dec 07;8(12):e1000557 [FREE Full text] [doi:10.1371/journal.pbio.1000557] [Medline: 21151886]
124. Neusser TP, Jörger KM, Schrödl M. Cryptic species in tropic sands--interactive 3D anatomy, molecular phylogeny andevolution of meiofaunal Pseudunelidae (Gastropoda, Acochlidia). PLoS One 2011;6(8):e23313 [FREE Full text] [doi:10.1371/journal.pone.0023313] [Medline: 21912592]
125. van den Berg G, Moorman AF. Development of the pulmonary vein and the systemic venous sinus: an interactive 3D
overview. PLoS One 2011;6(7):e22055 [FREE Full text] [doi: 10.1371/journal.pone.0022055] [Medline: 21779373]
126. Sizarov A, Ya J, de Boer BA, Lamers WH, Christoffels VM, Moorman AFM. Formation of the building plan of the humanheart: morphogenesis, growth, and differentiation. Circulation 2011 Mar 15;123(10):1125-1135 [FREE Full text] [doi:10.1161/CIRCULATIONAHA.110.980607] [Medline: 21403123]
127. Haszprunar G, Speimann E, Hawe A, Heß M. Interactive 3D anatomy and affinities of the Hyalogyrinidae, basal
Heterobranchia (Gastropoda) with a rhipidoglossate radula. Org Divers Evol 2011 Jun 19;11(3):201-236. [doi:10.1007/s13127-011-0048-0]
128. Ivanova EP, Hasan J, Truong VK, Wang JY, Raveggi M, Fluke C, et al. The influence of nanoscopically thin silver films
on bacterial viability and attachment. Appl Microbiol Biotechnol 2011 Aug;91(4):1149-1157. [doi:10.1007/s00253-011-3195-5] [Medline: 21556922]
129. Sylva M, Li VS, Buffing AA, van Es JH, van den Born M, van der Velden S, et al. The BMP antagonist follistatin-like 1
is required for skeletal and lung organogenesis. PLoS One 2011;6(8):e22616 [FREE Full text] [doi:10.1371/journal.pone.0022616] [Medline: 21826198]
130. Sizarov A, Devalla HD, Anderson RH, Passier R, Christoffels VM, Moorman AF. Molecular analysis of patterning ofconduction tissues in the developing human heart. Circ Arrhythm Electrophysiol 2011 Aug;4(4):532-542 [FREE Full text][doi: 10.1161/CIRCEP.111.963421] [Medline: 21576278]
131. Ivanova EP, Truong VK, Webb HK, Baulin VA, Wang JY, Mohammodi N, et al. Differential attraction and repulsion of
Staphylococcus aureus and Pseudomonas aeruginosa on molecularly smooth titanium films. Sci Rep 2011;1:165 [FREEFull text] [doi: 10.1038/srep00165] [Medline: 22355680]
132. Sizarov A, Lamers WH, Mohun TJ, Brown NA, Anderson RH, Moorman AF. Three-dimensional and molecular analysisof the arterial pole of the developing human heart. J Anat 2012 Apr;220(4):336-349 [FREE Full text] [doi:10.1111/j.1469-7580.2012.01474.x] [Medline: 22296102]
133. Brand J, Smith ES, Schwefel D, Lapatsina L, Poole K, Omerbašić D, et al. A stomatin dimer modulates the activity of
acid-sensing ion channels. EMBO J 2012 Aug 29;31(17):3635-3646 [FREE Full text] [doi: 10.1038/emboj.2012.203][Medline: 22850675]
134. Baeumler N, Haszprunar G, Ruthensteiner B. Development of the excretory system in a polyplacophoran mollusc: stages
in metanephridial system development. Front Zool 2012 Sep 14;9(1):23 [FREE Full text] [doi: 10.1186/1742-9994-9-23][Medline: 22973977]
135. Deans AR, Mikó I, Wipfler B, Friedrich F. Evolutionary phenomics and the emerging enlightenment of arthropod systematics.
Invert Systematics 2012;26(3):323. [doi: 10.1071/IS12063]
136. de Boer BA, van den Berg G, Soufan AT, de Boer PA, Hagoort J, van den Hoff MJ, et al. Measurement and 3D-visualizationof cell-cycle length using double labelling with two thymidine analogues applied in early heart development. PLoS One
2012;7(10):e47719 [FREE Full text] [doi: 10.1371/journal.pone.0047719] [Medline: 23091641]
137. Nakamura MJ, Terai J, Okubo R, Hotta K, Oka K. Three-dimensional anatomy of the Ciona intestinalis tailbud embryo atsingle-cell resolution. Dev Biol 2012 Dec 15;372(2):274-284 [FREE Full text] [doi: 10.1016/j.ydbio.2012.09.007] [Medline:
23022659]
138. Brenzinger B, Padula V, Schrödl M. Insemination by a kiss? Interactive 3D-microanatomy, biology and systematics of themesopsammic cephalaspidean sea slug Pluscula cuica Marcus, 1953 from Brazil (Gastropoda: Euopisthobranchia:
Philinoglossidae). Org Divers Evol 2012 Jun 12;13(1):33-54. [doi: 10.1007/s13127-012-0093-3]
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.115(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

139. Jensen B, van den Berg G, van den Doel R, Oostra RJ, Wang T, Moorman AF. Development of the hearts of lizards andsnakes and perspectives to cardiac evolution. PLoS One 2013;8(6):e63651 [FREE Full text] [doi:10.1371/journal.pone.0063651] [Medline: 23755108]
140. Brenzinger B, Haszprunar G, Schrödl M. At the limits of a successful body plan - 3D microanatomy, histology and evolutionof Helminthope (Mollusca: Heterobranchia: Rhodopemorpha), the most worm-like gastropod. Front Zool 2013 Jun
28;10(1):37 [FREE Full text] [doi: 10.1186/1742-9994-10-37] [Medline: 23809165]
141. Giraud-Billoud M, Gamarra-Luques C, Castro-Vazquez A. Functional anatomy of male copulatory organs of Pomacea
canaliculata (Caenogastropoda, Ampullariidae). Zoomorphology 2012 Dec 15;132(2):129-143. [doi:10.1007/s00435-012-0183-y]142. White MA, Benson RB, Tischler TR, Hocknull SA, Cook AG, Barnes DG, et al. New Australovenator hind limb elements
pertaining to the holotype reveal the most complete Neovenatorid leg. PLoS One 2013;8(7):e68649 [FREE Full text] [doi:10.1371/journal.pone.0068649] [Medline: 23894328]
143. Schulz-Mirbach T, Heß M, Metscher BD, Ladich F. A unique swim bladder-inner ear connection in a teleost fish revealed
by a combined high-resolution microtomographic and three-dimensional histological study. BMC Biol 2013 Jul 04;11:75[FREE Full text] [doi: 10.1186/1741-7007-11-75] [Medline: 23826967]
144. Schulz-Mirbach T, Heß M, Metscher BD. Sensory epithelia of the fish inner ear in 3D: studied with high-resolution contrast
enhanced microCT. Front Zool 2013 Oct 27;10(1):63 [FREE Full text] [doi: 10.1186/1742-9994-10-63] [Medline: 24160754]
145. Kohnert P, Brenzinger B, Jensen KR, Schrödl M. 3D- microanatomy of the semiterrestrial slug Gascoignella aprica Jensen,
1985—a basal plakobranchacean sacoglossan (Gastropoda, Panpulmonata). Org Divers Evol 2013 Jun 19;13(4):583-603.
[doi: 10.1007/s13127-013-0142-6]
146. Schulz-Mirbach T, Ladich F, Plath M, Metscher BD, Heß M. Are accessory hearing structures linked to inner ear morphology?
Insights from 3D orientation patterns of ciliary bundles in three cichlid species. Front Zool 2014 Mar 19;11(1):25 [FREEFull text] [doi: 10.1186/1742-9994-11-25] [Medline: 24645675]
147. Hawe A, Haszprunar G. 3D-microanatomy and histology of the hydrothermal vent gastropod Lurifax vitreus Warén &Bouchet, 2001 (Heterobranchia: Orbitestellidae) and comparisons with Ectobranchia. Org Divers Evol 2014;14:43-55. [doi:10.1007/s13127-013-0155-1]
148. Fernández R, Kvist S, Lenihan J, Giribet G, Ziegler A. Sine systemate chaos? A versatile tool for earthworm taxonomy:
non-destructive imaging of freshly fixed and museum specimens using micro-computed tomography. PLoS One
2014;9(5):e96617 [FREE Full text] [doi: 10.1371/journal.pone.0096617] [Medline: 24837238]
149. Cox PG, Faulkes CG. Digital dissection of the masticatory muscles of the naked mole-rat, Heterocephalus glaber (Mammalia,
Rodentia). PeerJ 2014;2:e448 [FREE Full text] [doi: 10.7717/peerj.448] [Medline: 25024917]
150. Sizarov A, de Bakker BS, Klein K, Ohlerth S. Building foundations for transcatheter intervascular anastomoses: 3D anatomyof the great vessels in large experimental animals. Interact Cardiovasc Thorac Surg 2014 Oct;19(4):543-551. [doi:10.1093/icvts/ivu210] [Medline: 24994699]
151. Lehmann T, Heß M, Wanner G, Melzer RR. Dissecting a neuron network: FIB-SEM-based 3D-reconstruction of the visualneuropils in the sea spider Achelia langi (Dohrn, 1881) (Pycnogonida). BMC Biol 2014 Aug 13;12:59 [FREE Full text][doi: 10.1186/s12915-014-0059-3] [Medline: 25285383]
152. Kucharzewski C, Raselimanana AP, Wang C, Glaw F. A taxonomic mystery for more than 150 years: Identity, systematic
position and Malagasy origin of the snake Elapotinus picteti Jan, 1862, and synonymy of Exallodontophis Cadle, 1999
(Serpentes: Lamprophiidae). Zootaxa 2014 Aug 14;3852(2):179-202. [Medline: 25284393]
153. Brenzinger B, Wilson NG, Schrödl M. Microanatomy of shelled Koloonella cf. minutissima (Laseron, 1951) (Gastropoda:‘lower’ Heterobranchia: Murchisonellidae) does not contradict a sister-group relationship with enigmatic Rhodopemorphaslugs. J Mollus Stud 2014 Jun 27;80(5):518-540. [doi: 10.1093/mollus/eyu036]154. Scherz MD, Ruthensteiner B, Vieites DR, Vences M, Glaw F. Two new microhylid frogs of the genus Rhombophryne withsuperciliary spines from the Tsaratanana Massif in Northern Madagascar. Herpetologica 2015 Dec;71(4):310-321. [doi:10.1655/HERPETOLOGICA-D-14-00048]
155. Schwaha TF, Handschuh S, Redl E, Wanninger A. Insights into the organization of plumatellid larvae (lophotrochozoa,
Bryozoa) by means of 3D-imaging and confocal microscopy. J Morphol 2015 Jan;276(1):109-120. [doi: 10.1002/jmor.20326][Medline: 25278218]
156. Grabe V, Strutz A, Baschwitz A, Hansson BS, Sachse S. Digital in vivo 3D atlas of the antennal lobe of Drosophilamelanogaster. J Comp Neurol 2015 Feb 15;523(3):530-544. [doi: 10.1002/cne.23697] [Medline: 25327641]
157. Sharp AC, Trusler PW. Morphology of the jaw-closing musculature in the common wombat (Vombatus ursinus) using
digital dissection and magnetic resonance imaging. PLoS One 2015;10(2):e0117730 [FREE Full text] [doi:10.1371/journal.pone.0117730] [Medline: 25707001]
158. Christensen CB, Lauridsen H, Christensen-Dalsgaard J, Pedersen M, Madsen PT. Better than fish on land? Hearing acrossmetamorphosis in salamanders. Proc Biol Sci 2015 Mar 07;282(1802) [FREE Full text] [doi: 10.1098/rspb.2014.1943][Medline: 25652830]
159. Wild E, Wollesen T, Haszprunar G, Heß M. Comparative 3D microanatomy and histology of the eyes and central nervous
systems in coleoid cephalopod hatchlings. Org Divers Evol 2014 Dec 12;15(1):37-64. [doi: 10.1007/s13127-014-0184-4]
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.116(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

160. Chen C, Copley JT, Linse K, Rogers AD, Sigwart JD. The heart of a dragon: 3D anatomical reconstruction of the 'scaly-footgastropod' (Mollusca: Gastropoda: Neomphalina) reveals its extraordinary circulatory system. Front Zool 2015;12:13 [FREEFull text] [doi: 10.1186/s12983-015-0105-1] [Medline: 26085836]
161. Kelder TP, Vicente-Steijn R, Harryvan TJ, Kosmidis G, Gittenberger-de Groot AC, Poelmann RE, et al. The sinus venosus
myocardium contributes to the atrioventricular canal: potential role during atrioventricular node development? J Cell Mol
Med 2015 Jun;19(6):1375-1389 [FREE Full text] [doi: 10.1111/jcmm.12525] [Medline: 25752780]
162. Soffers JH, Hikspoors JP, Mekonen HK, Koehler SE, Lamers WH. The growth pattern of the human intestine and its
mesentery. BMC Dev Biol 2015 Aug 22;15:31 [FREE Full text] [doi: 10.1186/s12861-015-0081-x] [Medline: 26297675]
163. Vicente-Steijn R, Scherptong RW, Kruithof BP, Duim SN, Goumans MJ, Wisse LJ, et al. Regional differences in WT-1
and Tcf21 expression during ventricular development: implications for myocardial compaction. PLoS One
2015;10(9):e0136025 [FREE Full text] [doi: 10.1371/journal.pone.0136025] [Medline: 26390289]
164. Bonsmann A, Stoffel MH, Burkhart M, Hatt JM. Anatomical atlas of the quail's ear (Coturnix coturnix). Anat Histol Embryol
2016 Oct;45(5):399-404. [doi: 10.1111/ahe.12198] [Medline: 26396065]
165. Razafimahatratra B, Wang C, Mori A, Glaw F. Potential envenomation by the aglyphous pseudoxyrhophiine snake
Leioheterodon madagascariensis and description of its dentition. J Venom Anim Toxins Incl Trop Dis 2015;21:47 [FREEFull text] [doi: 10.1186/s40409-015-0047-2] [Medline: 26594226]
166. van de Kamp T, Cecilia A, dos Santos Rolo T, Vagovič P, Baumbach T, Riedel A. Comparative thorax morphology ofdeath-feigning flightless cryptorhynchine weevils (Coleoptera: Curculionidae) based on 3D reconstructions. Arthropod
Struct Dev 2015 Nov;44(6):509-523. [doi: 10.1016/j.asd.2015.07.004] [Medline: 26259678]
167. Sumner-Rooney LH, Schrödl M, Lodde-Bensch E, Lindberg DR, Heß M, Brennan GP, et al. A neurophylogenetic approachprovides new insight to the evolution of Scaphopoda. Evol Dev 2015;17(6):337-346. [doi: 10.1111/ede.12164] [Medline:
26487042]
168. D'Amato G, Luxán G, del Monte-Nieto G, Martínez-Poveda B, Torroja C, Walter W, et al. Sequential Notch activation
regulates ventricular chamber development. Nat Cell Biol 2016 Jan;18(1):7-20 [FREE Full text] [doi: 10.1038/ncb3280][Medline: 26641715]
169. Loll B, Fabian H, Huser H, Hee CS, Ziegler A, Uchanska-Ziegler B, et al. Increased conformational flexibility of HLA-B*27
subtypes associated with ankylosing spondylitis. Arthritis Rheumatol 2016 Dec;68(5):1172-1182 [FREE Full text] [doi:10.1002/art.39567] [Medline: 26748477]
170. Felix PM, Gonçalves A, Vicente JR, Fonseca PJ, Amorim MC, Costa JL, et al. Optical micro-tomography “OPenT” allows
the study of large toadfish Halobatrachus didactylus embryos and larvae. Mech Dev 2016 Dec;140:19-24 [FREE Full text][doi: 10.1016/j.mod.2016.03.001] [Medline: 27000637]
171. Rybak J, Talarico G, Ruiz S, Arnold C, Cantera R, Hansson BS. Synaptic circuitry of identified neurons in the antennal
lobe of Drosophila melanogaster. J Comp Neurol 2016 Jun 15;524(9):1920-1956. [doi: 10.1002/cne.23966] [Medline:
26780543]
172. Fischer G, Sarnat EM, Economo EP. Revision and microtomography of the Pheidole knowlesi group, an endemic ant
radiation in Fiji (Hymenoptera, Formicidae, Myrmicinae)Myrmicinae). PLoS One 2016;11(7):e0158544 [FREE Full text][doi: 10.1371/journal.pone.0158544] [Medline: 27462877]
173. Haszprunar G, Kunze T, Brückner M, Heß M. Towards a sound definition of Skeneidae (Mollusca, Vetigastropoda): 3D
interactive anatomy of the type species, Skenea serpuloides (Montagu, 1808) and comments on related taxa. Org Divers
Evol 2016 Jan 14;16(3):577-595. [doi: 10.1007/s13127-015-0260-4]
174. Neusser TP, Jörger KM, Lodde-Bensch E, Strong EE, Schrödl M. The unique deep sea-land connection: interactive 3D
visualization and molecular phylogeny ofn. sp. (Bathyhedylidae n. fam.)-the first panpulmonate slug from bathyal zones.
PeerJ 2016;4:e2738 [FREE Full text] [doi: 10.7717/peerj.2738] [Medline: 27957391]
175. Scherz MD, Hawlitschek O, Andreone F, Rakotoarison A, Vences M, Glaw F. A review of the taxonomy and osteologyof the Rhombophryne serratopalpebrosa species group (Anura: Microhylidae) from Madagascar, with comments on thevalue of volume rendering of micro-CT data to taxonomists. Zootaxa 2017 Jun 06;4273(3):301-340. [doi:10.11646/zootaxa.4273.3.1]
176. Wu Y, Dabhoiwala NF, Hagoort J, Tan LW, Zhang SX, Lamers WH. Architectural differences in the anterior and middle
compartments of the pelvic floor of young-adult and postmenopausal females. J Anat 2017 May;230(5):651-663. [doi:10.1111/joa.12598] [Medline: 28299781]
177. Lambert SM, Hutter CR, Scherz MD. Diamond in the rough: a new species of fossorial diamond frog (Rhombophryne)
from Ranomafana National Park, southeastern Madagascar. ZSE 2017 Feb 24;93(1):143-155. [doi: 10.3897/zse.93.10188]
178. Chen C, Uematsu K, Linse K, Sigwart JD. By more ways than one: Rapid convergence at hydrothermal vents shown by
3D anatomical reconstruction of Gigantopelta (Mollusca: Neomphalina). BMC Evol Biol 2017 Dec 01;17(1):62 [FREEFull text] [doi: 10.1186/s12862-017-0917-z] [Medline: 28249568]
179. Klinkhamer AJ, Wilhite DR, White MA, Wroe S. Digital dissection and three-dimensional interactive models of limb
musculature in the Australian estuarine crocodile (Crocodylus porosus). PLoS One 2017;12(4):e0175079 [FREE Full text][doi: 10.1371/journal.pone.0175079] [Medline: 28384201]

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.117(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

180. Herzog H, Klein B, Ziegler A. Form and function of the teleost lateral line revealed using three-dimensional imaging andcomputational fluid dynamics. J R Soc Interface 2017 May;14(130). [doi: 10.1098/rsif.2016.0898] [Medline: 28468922]
181. Smith TD, McMahon MJ, Millen ME, Llera C, Engel SM, Li L, et al. Growth and development at the sphenoethmoidal
junction in perinatal primates. Anat Rec (Hoboken) 2017 Dec;300(12):2115-2137. [doi: 10.1002/ar.23630] [Medline:
28667704]
182. Dreyer N, Høeg JT, Heß M, Sørensen S, Spremberg U, Yusa Y. When dwarf males and hermaphrodites copulate: firstrecord of mating behaviour in a dwarf male using the androdioecious barnacle Scalpellum scalpellum (Crustacea: Cirripedia:
Thoracica). Org Divers Evol 2018 Mar;18(1):115-123. [doi: 10.1007/s13127-017-0349-z]
183. Fredericksen MA, Zhang Y, Hazen ML, Loreto RG, Mangold CA, Chen DZ, et al. Three-dimensional visualization and a
deep-learning model reveal complex fungal parasite networks in behaviorally manipulated ants. Proc Natl Acad Sci U S
A 2017 Nov 21;114(47):12590-12595 [FREE Full text] [doi: 10.1073/pnas.1711673114] [Medline: 29114054]
184. de Bakker BS, de Bakker HM, Soerdjbalie-Maikoe V, Dikkers FG. The development of the human hyoid-larynx complex
revisited. Laryngoscope 2017 Dec 08. [doi: 10.1002/lary.26987] [Medline: 29219191]
185. Betts MW, Maschner HD, Schou CD, Schlader R, Holmes J, Clement N, et al. Virtual zooarchaeology: building a web-basedreference collection of northern vertebrates for archaeofaunal research and education. Journal of Archaeological Science
2011 Apr;38(4):755.e1-755.e9. [doi: 10.1016/j.jas.2010.06.021]
186. Pingault JB, Côté SM, Lacourse E, Galéra C, Vitaro F, Tremblay RE. Childhood hyperactivity, physical aggression andcriminality: a 19-year prospective population-based study. PLoS One 2013;8(5):e62594 [FREE Full text] [doi:10.1371/journal.pone.0062594] [Medline: 23658752]
187. Poelmann RE, Gittenberger-de Groot AC, Vicente-Steijn R, Wisse LJ, Bartelings MM, Everts S, et al. Evolution anddevelopment of ventricular septation in the amniote heart. PLoS One 2014;9(9):e106569 [FREE Full text] [doi:10.1371/journal.pone.0106569] [Medline: 25192012]
188. Li Z, Clarke JA. New insight into the anatomy of the hyolingual apparatus of Alligator mississippiensis and implications
for reconstructing feeding in extinct archosaurs. J Anat 2015 Jul;227(1):45-61 [FREE Full text] [doi: 10.1111/joa.12320][Medline: 26018316]
189. Charles JP, Cappellari O, Spence AJ, Hutchinson JR, Wells DJ. Musculoskeletal geometry, muscle architecture and functional
specialisations of the mouse hindlimb. PLoS One 2016;11(4):e0147669 [FREE Full text] [doi: 10.1371/journal.pone.0147669][Medline: 27115354]
190. Porter WR, Sedlmayr JC, Witmer LM. Vascular patterns in the heads of crocodilians: blood vessels and sites of thermal
exchange. J Anat 2016 Dec;229(6):800-824. [doi: 10.1111/joa.12539] [Medline: 27677246]
191. Shin DS, Shim YJ, Kim BC. Sectioned images and surface models of a cadaver head with reference to botulinum neurotoxin
injection. Folia Morphol (Warsz) 2018 Jan 18 [FREE Full text] [doi: 10.5603/FM.a2018.0005] [Medline: 29345721]
192. Husch A, Petersen MV, Gemmar P, Goncalves J, Sunde N, Hertel F. Post-operative deep brain stimulation assessment:
Automatic data integration and report generation. Brain Stimul 2018 Feb 01 [FREE Full text] [doi: 10.1016/j.brs.2018.01.031][Medline: 29429953]
193. Shin DS, Shim YJ, Kim BC. Sectioned images and 3D models of a cadaver head with reference to dermal filler injection.
Ann Anat 2018 May;217:34-39. [doi: 10.1016/j.aanat.2018.02.001] [Medline: 29481857]
194. Bicknell RD, Klinkhamer AJ, Flavel RJ, Wroe S, Paterson JR. A 3D anatomical atlas of appendage musculature in thechelicerate arthropod Limulus polyphemus. PLoS One 2018;13(2):e0191400 [FREE Full text] [doi:10.1371/journal.pone.0191400] [Medline: 29444161]
195. van Wijk B, van den Berg G, Abu-Issa R, Barnett P, van der Velden S, Schmidt M, et al. Epicardium and myocardium
separate from a common precursor pool by crosstalk between bone morphogenetic protein- and fibroblast growth
factor-signaling pathways. Circ Res 2009 Aug 28;105(5):431-441 [FREE Full text] [doi: 10.1161/CIRCRESAHA.109.203083][Medline: 19628790]
196. Shin DS, Jang HG, Park JS, Park HS, Lee S, Chung MS. Accessible and informative sectioned images and surface models
of a cadaver head. J Craniofac Surg 2012 Jul;23(4):1176-1180. [doi: 10.1097/SCS.0b013e31825657d8] [Medline: 22801119]
197. Prats-Galino A, Reina MA, Mavar-Haramija M, Puigdellivol-Sánchez A, Juanes Mendez JA, de Andres JA. 3D interactivemodel of lumbar spinal structures of anesthetic interest. Clin Anat 2015 Mar;28(2):205-212. [doi: 10.1002/ca.22479][Medline: 25352014]
198. Shin DS, Jang HG, Hwang SB, Har DH, Moon YL, Chung MS. Two-dimensional sectioned images and three-dimensionalsurface models for learning the anatomy of the female pelvis. Anat Sci Educ 2013;6(5):316-323. [doi: 10.1002/ase.1342][Medline: 23463707]
199. Chung BS, Ahn YH, Park JS. Ten triangles around cavernous sinus for surgical approach, described by schematic diagram
and three dimensional models with the sectioned images. J Korean Med Sci 2016 Sep;31(9):1455-1463 [FREE Full text][doi: 10.3346/jkms.2016.31.9.1455] [Medline: 27510391]
200. de Bakker BS, de Jong KH, Hagoort J, de Bree K, Besselink CT, de Kanter FE, et al. An interactive three-dimensional
digital atlas and quantitative database of human development. Science 2016 Dec 25;354(6315):aag0053. [doi:10.1126/science.aag0053] [Medline: 27884980]

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.118(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

201. Park HS, Chung MS, Shin DS, Jung YW, Park JS. Accessible and informative sectioned images, color-coded images, andsurface models of the ear. Anat Rec (Hoboken) 2013 Aug;296(8):1180-1186 [FREE Full text] [doi: 10.1002/ar.22719][Medline: 23713007]
202. Park HS, Shin DS, Cho DH, Jung YW, Park JS. Improved sectioned images and surface models of the whole dog body.
Ann Anat 2014 Sep;196(5):352-359. [doi: 10.1016/j.aanat.2014.05.036] [Medline: 24986152]
203. Kim BC, Chung MS, Park HS, Shin DS, Park JS. Accessible and informative sectioned images and surface models of themaxillofacial area for orthognathic surgery. Folia Morphol (Warsz) 2015;74(3):346-351. [doi: 10.5603/FM.2015.0052][Medline: 26339816]
204. Shin DS, Lee S, Park HS, Lee SB, Chung MS. Segmentation and surface reconstruction of a cadaver heart on Mimics
software. Folia Morphol (Warsz) 2015;74(3):372-377. [doi: 10.5603/FM.2015.0056] [Medline: 26339820]
205. Shin DS, Kim HJ, Kim BC. Sectioned images and surface models of a cadaver for understanding the dorsalis pedis flap. J
Craniofac Surg 2015 Jul;26(5):1656-1659. [doi: 10.1097/SCS.0000000000001618] [Medline: 26079120]
206. Park JS, Jung YW. Software for browsing sectioned images of a dog body and generating a 3D model. Anat Rec (Hoboken)
2016 Jan;299(1):81-87 [FREE Full text] [doi: 10.1002/ar.23200] [Medline: 26219434]
207. Reina MA, Lirk P, Puigdellívol-Sánchez A, Mavar M, Prats-Galino A. Human lumbar ligamentum flavum anatomy for
epidural anesthesia: reviewing a 3D MR-based interactive model and postmortem samples. Anesth Analg 2016
Mar;122(3):903-907. [doi: 10.1213/ANE.0000000000001109] [Medline: 26891398]
208. Shin DS, Kim HJ, Kim BC. Sectioned images and surface models of a cadaver for understanding the free vascularised
anterior rib flap. Folia Morphol (Warsz) 2017;76(1):117-122. [doi: 10.5603/FM.a2016.0035] [Medline: 27830889]
209. Jang HG, Chung MS, Shin DS. Portable Document Format file containing the surface models to learn the stereoscopic
shape of foot muscles. Int J Morphol 2015 Dec;33(4):1287-1292. [doi: 10.4067/S0717-95022015000400016]
210. Shin DS, Chung MS. Virtual movement of the ankle and subtalar joints using cadaver surface models. Int J Morphol 2015
Sep;33(3):888-894. [doi: 10.4067/S0717-95022015000300014]
211. Chung BS, Shin DS, Brown P, Choi J, Chung MS. Virtual dissection table including the visible Korean images, complemented
by free software of the same data. Int J Morphol 2015 Jun;33(2):440-445. [doi: 10.4067/S0717-95022015000200006]
212. Park HS, Chung MS, Shin DS, Jung YW, Park JS. Whole courses of the oculomotor, trochlear, and abducens nerves,
identified in sectioned images and surface models. Anat Rec (Hoboken) 2015 Feb;298(2):436-443 [FREE Full text] [doi:10.1002/ar.23048] [Medline: 25212480]
213. Shin DS, Kwon K, Shin B, Park HS, Lee S, Lee S, et al. Surface models and gradually stripped volume model to explore
the foot muscles. Anatomy 2015 Apr 30;9(1):19-25. [doi: 10.2399/ana.14.052]
214. Chung BS, Chung MS, Park JS. Six walls of the cavernous sinus identified by sectioned images and three-dimensionalmodels: anatomic report. World Neurosurg 2015 Aug;84(2):337-344. [doi: 10.1016/j.wneu.2015.03.049] [Medline: 25839400]
215. Chung BS, Park JS, Jang HG, Chung MS. Software to browse the pictures of two knees in diverse states of dissection,
flexion and rotation. Int J Morphol 2015 Sep;33(3):1009-1015. [doi: 10.4067/S0717-95022015000300032]
216. Chung BS, Chung MS, Park HS, Shin B, Kwon K. Colonoscopy tutorial software made with a cadaver's sectioned images.
Ann Anat 2016 Nov;208:19-23. [doi: 10.1016/j.aanat.2016.06.010] [Medline: 27475426]
217. Chung BS, Kwon K, Shin BS, Chung MS. Surface models and gradually peeled volume model to explore hand structures.
Ann Anat 2017 May;211:202-206. [doi: 10.1016/j.aanat.2017.02.002] [Medline: 28274804]
218. Park HS, Choi DH, Park JS. Improved sectioned images and surface models of the whole female body. Int J Morphol 2015
Dec;33(4):1323-1332. [doi: 10.4067/S0717-95022015000400022]
219. Chung BS, Chung MS, Shin BS, Kwon K. Three software tools for viewing sectional planes, volume models, and surface
models of a cadaver hand. J Korean Med Sci 2018 Feb 19;33(8):e64 [FREE Full text] [doi: 10.3346/jkms.2018.33.e64][Medline: 29441756]
220. Valera-Melé M, Puigdellívol-Sánchez A, Mavar-Haramija M, Juanes-Méndez JA, San-Román L, de Notaris M, et al. A
novel and freely available interactive 3D model of the internal carotid artery. J Med Syst 2018 Mar 05;42(4):72. [doi:10.1007/s10916-018-0919-4] [Medline: 29508089]
221. Warmann SW, Schenk A, Schaefer JF, Ebinger M, Blumenstock G, Tsiflikas I, et al. Computer-assisted surgery planningin children with complex liver tumors identifies variability of the classical Couinaud classification. J Pediatr Surg 2016
Nov;51(11):1801-1806. [doi: 10.1016/j.jpedsurg.2016.05.018] [Medline: 27289416]
222. Barnes DG, Fluke CJ, Bourke PD, Parry OT. An advanced, three-dimensional plotting library for astronomy. Publ Astron
Soc Aust 2013 Mar 5;23(02):82-93. [doi: 10.1071/AS06009]
223. Swinburne University of Technology. S2PLOT: A three-dimensional plotting library URL: http://astronomy.swin.edu.au/
s2plot/index.php?title=S2PLOT [accessed 2018-03-03] [WebCite Cache ID 6xdskoYtn]
224. The LaTeX Project. LaTeX - A document preparation system URL: https://www.latex-project.org [accessed 2018-03-03][WebCite Cache ID 6xdseFf7s]
225. Grahn A. CTAN Comprehensive Tex Archive Network. movie15 – Multimedia inclusion package URL: https://ctan.org/
pkg/movie15 [accessed 2018-03-03] [WebCite Cache ID 6xdsVN39p]
226. Asymptote. Asymptote - SourceForge URL: http://asymptote.sourceforge.net [accessed 2018-03-03] [WebCite Cache ID
6xdstUnRp]
http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.119(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Newe & Becker

227. Haru free PDF library. URL: http://libharu.org/ [accessed 2018-03-03] [WebCite Cache ID 6xdt1s7rK]
228. Ritter F, Boskamp T, Homeyer A, Laue H, Schwier M, Link F, et al. Medical image analysis. IEEE Pulse 2011
Nov;2(6):60-70. [doi: 10.1109/MPUL.2011.942929] [Medline: 22147070]
229. MeVis Medical Solutions AG and Fraunhofer MEVIS. MeVisLab. URL: https://www.mevislab.de/ [accessed 2018-03-03][WebCite Cache ID 6xduIVHna]
230. de Bakker BS. 3D Atlas of Human Embryology. URL: https://www.3dembryoatlas.com/ [accessed 2018-03-03] [WebCite
Cache ID 6xdtZCf9G]
231. KISTI. Visible Korean. Overview of the Visible Korean Project URL: http://vkh3.kisti.re.kr/?q=node/24 [accessed
2018-03-03] [WebCite Cache ID 6xdtfOivE]
232. Springob CM, Magoulas C, Colless M, Mould J, Erdogdu P, Jones DH, et al. The 6dF Galaxy Survey: peculiar velocity
field and cosmography. Mon Not R Astron Soc 2014;445(3):2677-2697. [doi: 10.1093/mnras/stu1743]
233. Lautenschlager S, Butler RJ. Neural and endocranial anatomy of Triassic phytosaurian reptiles and convergence with fossil
and modern crocodylians. PeerJ 2016;4:e2251 [FREE Full text] [doi: 10.7717/peerj.2251] [Medline: 27547557]
234. Zanno LE, Drymala S, Nesbitt SJ, Schneider VP. Early crocodylomorph increases top tier predator diversity during rise ofdinosaurs. Sci Rep 2015 Mar 19;5:9276 [FREE Full text] [doi: 10.1038/srep09276] [Medline: 25787306]
235. Vasilyev V. Towards interactive 3D graphics in chemistry publications. Theor Chem Acc 2010;125(3-6):173-176. [doi:10.1007/s00214-009-0636-7.x]

Abbreviations
2D: two-dimensional
3D: three-dimensional
ECMA: European Computer Manufacturers Association
ISO: International Organization for Standardization
MiB: mebibyte, (binary) megabytes
PDF: Portable Document Format
PMI: product manufacturing information
PRC: Product Representation Compact
U3D: Universal 3D

Edited by G Eysenbach; submitted 03.03.18; peer-reviewed by M Puttaroo, J Veenland; comments to author 26.04.18; revised version
received 07.05.18; accepted 24.05.18; published 07.08.18.
Please cite as:
Newe A, Becker L
Three-Dimensional Portable Document Format (3D PDF) in Clinical Communication and Biomedical Sciences: Systematic Review
of Applications, Tools, and Protocols
JMIR Med Inform 2018;6(3):e10295
URL: http://medinform.jmir.org/2018/3/e10295/
doi:10.2196/10295
PMID:30087092©Axel Newe, Linda Becker. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 07.08.2018. This is
an open-access article distributed under the terms of the Creative Commons Attribution License(https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information,a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.

http://medinform.jmir.org/2018/3/e10295/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 |e10295 | p.120(page number not for citation purposes)

Publisher:
JMIR Publications
130 Queens Quay East.
Toronto, ON, M5A 3Y5
Phone: (+1) 416-583-2040
Email: support@jmir.org
https://www.jmirpublications.com/

XSL• FO
RenderX