JMIR MEDICAL INFORMATICS

Graham et al

Original Paper

Emergency Physician Use of the Alberta Netcare Portal, aProvince-Wide Interoperable Electronic Health Record:
Multi-Method Observational Study
Timothy AD Graham1,2, MD; Mark Ballermann1,3, PhD; Eddy Lang4, MD; Michael J Bullard2, MD; Denise Parsons5,
MSc; Gabriella Mercuur5, MD; Pat San Agustin2, MD; Samina Ali5, MD
1

Alberta Health Services, Edmonton, AB, Canada

2

Department of Emergency Medicine, University of Alberta, Edmonton, AB, Canada

3

Department of Critical Care Medicine, University of Alberta, Edmonton, AB, Canada

4

Department of Emergency Medicine, University of Calgary, Calgary, AB, Canada

5

Department of Paediatrics, University of Alberta, Edmonton, AB, Canada

Corresponding Author:
Timothy AD Graham, MD
Alberta Health Services
5th Floor, 44 Capital Boulevard
10044-108 Street
Edmonton, AB, T5J 3S7
CanadaPhone: 1 780 431 2549
Fax: 1 7806139804
Email: tim.graham@ahs.ca

Abstract
Background: The adoption and use of an electronic health record (EHR) can facilitate real-time access to key health informationand support improved outcomes. Many Canadian provinces use interoperable EHRs (iEHRs) to facilitate health informationexchange, but the clinical use and utility of iEHRs has not been well described.
Objective: The aim of this study was to describe the use of a provincial iEHR known as the Alberta Netcare Portal (ANP) in
4 urban Alberta emergency departments. The secondary objectives were to characterize the time spent using the respective
electronic tools and identify the aspects that were perceived as most useful by emergency department physicians.
Methods: In this study, we have included 4 emergency departments, 2 using paper-based ordering (University of Alberta
Hospital [UAH] and Grey Nuns Community Hospital [GNCH]) and 2 using a commercial vendor clinical information system(Peter Lougheed Centre [PLC] and Foothills Medical Centre [FMC]). Structured clinical observations of ANP use and system
audit logs analysis were compared at the 4 sites from October 2014 to March 2016.
Results: Observers followed 142 physicians for a total of 566 hours over 376 occasions. The median percentage of observed
time spent using ANP was 8.5% at UAH (interquartile range, IQR, 3.7%-13.3%), 4.4% at GNCH (IQR 2.4%-4.4%), 4.6% at
FMC (IQR 2.4%-7.6%), and 5.1% at PLC (IQR 3.0%-7.7%). By combining administrative and access audit data, the median
number of ANP screens (ie, results and reports displayed on a screen) accessed per patient visit were 20 at UAH (IQR 6-67), 9
at GNCH (IQR 4-29), 7 at FMC (IQR 2-18), and 5 at PLC (IQR 2-14). When compared with the structured clinical observations,the statistical analysis of screen access data showed that ANP was used more at UAH than the other sites.
Conclusions: This study shows that the iEHR is well utilized at the 4 sites studied, and the usage patterns implied clinical value.
Use of the ANP was highest in a paper-based academic center and lower in the centers using a commercial emergency department
clinical information system. More study about the clinical impacts of using iEHRs in the Canadian context including longer term
impacts on quality of practice and safety are required.
(JMIR Med Inform 2018;6(3):e10184) doi: 10.2196/10184

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

KEYWORDS
ambulatory care facilities; cross-sectional studies; electronic health records; utilization; hospital emergency service; statistics andnumerical data; health information exchange; humans; information dissemination; medical record linkage; program evaluation

Introduction

department. This trend was seen early in the deployment of the
ANP [18].

Background

The ANP was introduced in 2004 in 13 hospitals, 22 publichealth centers, 9 mental health clinics, and a small number ofprivate physician offices within the Edmonton Zone of Alberta
Health Services (formerly known as Capital Health Authority)
and was rapidly adopted by providers in diverse careenvironments [17,18].

The adoption and use of an electronic health record (EHR) can
facilitate real-time access to key health information and support
improved outcomes [1-9]. Canadian hospitals are still primarily
paper based, with only 5.2% of hospitals progressing to stage
five or higher on the 7-point Electronic Medical Record
Adoption Scale [10]. Although most hospital-based care is still
documented on paper, there has been systematic investment
and adoption of digital ancillary systems such as laboratory,
pharmacy, and diagnostic imaging, sponsored by Canada Health
Infoway (Infoway). Infoway is an independent, not-for-profit
organization funded by the federal government, which works
with the provinces and territories to cofund digital healthprojects throughout Canada.
One of the Canadian strategies for digitizing health care has
been the promotion of the interoperable EHR (iEHR), which isequivalent to the health information exchange concept used inother jurisdictions [11,12]. iEHRs are being developed by eachprovince as a part of the wider Canadian initiative to connect
health care nationally. iEHRs are intended to be a longitudinalsummary of key health events and act as a complement topoint-of-service systems such as the electronic medical record(EMR) [13]. iEHRs have been shown to reduce duplicativelaboratory and radiology testing, emergency department costs,
and hospital admissions, with most clinicians also attributing
positive changes in care coordination, communication, andknowledge about patients [14]. In contrast to the iEHR, in
Canada, EMRs tend to be facility centric and refer to an
ambulatory care record primarily linked to one single-care
environment (such as a particular physician’s office) or ahospital-based clinical information system.
In Canada, 12 of the 13 provincial and territorial governments
have implemented iEHRs, but their maturity is quite variable.
To date, only 5 of the 12 have made the 4 planned clinicalcomponents available, including diagnostic images, laboratory
test results, dispensed drugs, clinical reports, and immunizations[15].
Alberta (population 4.2 million) was the first Canadian province
to develop a provincial read-only iEHR known as the Alberta
Netcare Portal (ANP). The ANP iEHR provides read-only accessto province-wide laboratory results, diagnostic imaging (reports
and images), electrocardiograms, textual reports (dictateddischarge summaries, operative reports, some consultations,etc), scanned reports (including the paper emergency department
charts from the Edmonton Zone), dispensed drug information
via the provincial Pharmaceutical Information Network, andcontains data from more clinical domains than any otherprovince [16,17]. ANP has become an important tool forcontinuity of care, and although inpatient care is provided on
paper charts, the information in the ANP has almost entirely
supplanted the need to refer to paper charts in the emergency
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

In 2008, there was a substantial reorganization of health carein Alberta, and multiple health regions were amalgamated into
the first Canadian provincial Health Authority called Alberta
Health Services (AHS). Currently, AHS comprises 5 functionaland regional zones, each with varying degrees of health caredigitization. In acute care facilities, including emergency
departments, the vast majority of the Edmonton Zone uses paperfor documentation and order entry, with digital radiology as
well as electronic patient tracking and laboratory resultsdelivery. In contrast, the Calgary Zone uses an emergency
department and inpatient clinical information system from acommercial vendor (Allscripts), with virtually 100%
computerized provider order entry. The ANP is available as atab and launched from within the Allscripts interface.
Currently, ANP is used in a read-only fashion in 111 hospitals
and more than 650 facilities affiliated within AHS, as well asfamily practice clinics, pharmacies, and other health care entities
in the province. In 2018, over 51,000 physicians, nurses,
pharmacists, and other Alberta health care providers hadrole-based ANP access, with more than 1.9 million patient
records accessed monthly [19]. The ANP’s evolution from a
regional to a province-wide system and the usage patterns andcontrasting information systems between zones in AHS
presented a natural experiment and an excellent opportunity to
study the ANP’s utility and usage in 4 busy urban emergency
departments.

Objective
The primary objective of this study was to compare the use of
an iEHR (health information exchange-type) called ANP andother ancillary electronic systems between emergency
department physicians practicing primarily in paper-based
emergency departments versus clinical information system-based
emergency departments. The secondary objectives were tocharacterize the time spent using the respective electronic tools
and identify the aspects that were perceived as most useful by
emergency department physicians.

Methods
Setting and Participants
We included 4 emergency departments in the study, 2 in the
Edmonton Zone (University of Alberta Hospital [UAH] andGrey Nuns Community Hospital [GNCH]) and 2 in the Calgary
Zone (Peter Lougheed Centre [PLC] and Foothills Medical
JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Centre [FMC]). Data were collected between October 2014 andMarch 2016. The UAH and FMC are comparably large academic
tertiary care facilities and the GNCH and PLC are comparable
community hospitals. The mean number of emergency
department visits annually was 66,003 (UAH), 67,300 (GNCH),
80,466 (FMC), and 79,172 (PLC) during the period of the study.
All physicians providing care in at least one of the 4 study sites
were eligible for inclusion in the direct observation andinterview portions of the study. All uses of the ANP related tothe patients presenting to the emergency department of any
study hospital during the defined study period (October 2014
to March 31, 2016) were considered eligible for inclusion in
the administrative database record review.

Emergency Department Information Systems andTools
The Edmonton Zone and Calgary Zone emergency departments
use different mixes of paper and electronic information systems
to provide care and manage health information (Table 1). Patient
tracking and emergency department lab results in Edmonton
Zone are available on a commercial emergency department
information system, but all order entry and charting is done on
paper. Diagnostic images are available on stand-alone picturearchiving and communication system stations that are notintegrated with the emergency department information system.
Historical health information in Edmonton is usually obtained
via the ANP, which is accessed through a Web browser.
During the study period, care documentation in the Calgary
Zone emergency departments was provided using a paper chart.
Orders were managed using computerized provider order entry
entered into Allscripts, which also provided the relevant
laboratory, diagnostic imaging, and pharmaceutical information
primarily related to Calgary Zone. ANP is available as aparameter-based launch from within the Allscripts interface.
As Allscripts in Calgary contains most relevant patient data, it
was hypothesized that the use of ANP would be much less in
the Calgary zone and primarily related to patient care that
occurred outside the Calgary Zone.

Ethics and Approvals
Research ethics approval was obtained from both the Universityof Alberta (Approval Pro00033249) and the University ofCalgary (Approval REB13-0204) for the observation andinterview portions of the study. The database review including
the ANP access audit data was obtained following guidelines
from provincial legislation that permits the use of data forsystem quality improvement purposes, which was the primary
goal for this data extraction and analysis. The A pRoject Ethics
Community Consensus Initiative screening tool was used as a
guide in this process, and the resulting risk was assessed as
being low.

Structured Clinical Observations
Participants
To prepare for the study, emergency department administrators,
physicians, nurses, and allied health staff were informed about
the study using email notices, visits to departmental meetings,
and posted notices in the emergency departments. Physicians
were approached by observers during their regularly scheduled
shift. Of the 151 physicians approached regarding participation,
146 provided informed consent and 5 declined to participate.
At the initial encounter, physicians provided demographic data,
including their age, sex, payment model, comfort with computers(Likert scale 0-10), comfort interacting with EMR (Likert scale0-10), number of years in practice, and medical credentials.

Procedure
Observations were conducted following previously publishedmethods and using the Work Observation Method by Activity
Timing (WOMBAT) software (WOMBAT 2.0, University of
New South Wales) [20,21]. Research assistants were trained in
the study protocol, use of the tablet-based WOMBAT datacollection tool, and the emergency department patient-tracking
system. The 5 research assistants were trained for at least 12
hours before starting observations. Training also consisted offamiliarization with the data collection tool and data definitions(Multimedia Appendix 1), then completion of an observation
alongside a more experienced observer.

Table 1. Overview of functions available using different applications at the different sites.
Information system function

Application
Emergency Department
Information SystemaaAlberta Netcare Portalb

Sunrise Clinical Manager

Picture Archiving andClinical Information Systemc Communication Systemb

Track patientsYes

Not in real time

Yes

No

View lab results

Yes

Yes

Yes

No

View diagnostic imaging resultsNo

Yes (images and reports)

Yes (images and reports)

Yes (images and reports)

Order medications

No

No

Yes

No

Clinical documentation

No

Text reports (eg, consult
letters)No

No

Handover typed note

Yes

No

Yes

No

Edmonton.

b

Calgary and Edmonton.

c

Calgary.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
One of the authors (MB) was responsible for training research
assistants and ensured that research assistants achieved
inter-rater agreement scores of at least 90% with respect to thetime spent on tasks and numbers of different tasks recorded
before completing solo observations. Each research assistantconducted observations across multiple sites.
The research assistants approached emergency department
physicians during the earlier hours of their shift, as this was
posited to be when physicians would be more actively involved
in direct clinical care, rather than the end of the shift whensign-overs and charting were being completed. After obtaining
consent, the research assistants observed the physician for 90
min of their clinical shift and recorded demographic and clinicalinformation regarding each patient seen and the time andclinician reason for each clinical access of ANP or otherinformation systems. Time-stamped records including the tasks
being completed, the people who were present, and which
information tools were in use were recorded for later analysis.
Observations were balanced for the time of day (morning 8 am
to 12 pm, afternoon 12 pm to 4 pm, and evening 4 pm to 8 pm)
and time of week (midweek, weekend, Monday, and Friday).
Qualitative field notes supported the capture of additional
contextual information, including the number of learners(medical students and residents) on shift with the physician, thebusyness of physician shift, type of patients seen (ie, chronic,
complex vs ambulatory, pediatric vs geriatric), and type ofpresentation, when possible. Inter-rater reliability was calculated
during training sessions where learning observers were trained
to score the activities alongside more experienced observers,
including the second author. Observers conducted independent
data collection after consistently scoring 90% or higher
agreement on participant time spent on patient care tasks.

Administrative Data Matching and Review
Administrative data for visits to the 4 sites, including the lengthof stay, final disposition (inpatient admission or discharge), anddischarge diagnosis were obtained for October 2014 to March2016. These data are routinely recorded as a part of standard
AHS operations and for auditing provider accesses to patient
records. emergency department encounters were identified froma national system known as the National Ambulatory Care
Reporting System (NACRS) [22], and data matching was
conducted using the available patient identifiers in both the
NACRS and the ANP access audit databases. The number ofscreens viewed on the day of an emergency department
encounter and the subsequent day was retrieved from the accessaudit data. The 2 datasets were matched using a common Unique
Lifetime Identifier, a provincially assigned specific number
used for patient identification purposes. Screen views were
included in the analysis if they happened on either the day ofthe emergency department visit or the subsequent day. A screen
view was defined by a change in the context of the informationthat an emergency department physician accessed on the ANP(Multimedia Appendix 2). For example, after searching for andidentifying a patient, the first screen is a demographics screenwith a Clinical Document Viewer with links to various typesof clinical information. Clicking on a link might provide a single
lab value, a lab flow sheet with multiple values, a textual report,
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

Graham et ala diagnostic image, or an electrocardiogram, each on a different
screen view within the ANP, and each counting as a separate
access.

Analysis
Quantitative analysis was completed using R (3.4.3, The
R-Foundation) and Tableau (10.3). Simple linear regression
was completed to evaluate potential effects of demographic,
practice, and site-related explanatory variables on the percentageof time spent using ANP and other emergency department
information systems tools such as emergency department
information system and picture archiving and communication
system. Additionally, structured physician observation records
included contextual notes to ascertain what information needs
were being met when accessing information tools duringobservations. Differences between sites were evaluated usinga Kruskal-Wallis analysis of variance. The initial study questions
were used to develop the semistructured questionnaire toevaluate what value clinicians found in the tools they used,
including the iEHR, which were evaluated against thequantitative results to support triangulation of the findings.

Results
Structured Clinical Observations
To evaluate the context of emergency department physician useof ANP during clinical work, research assistants recorded
structured clinical observations at the 4 research emergency
departments. Between October 2014 and July 2015, 376
structured, 90-min clinical observations were completed with
142 physician participants (Sites UAH=97, GNCH=94,
FMC=99, and PLC=83). As emergency department physiciansin Calgary provide care across the different sites in each city,
26 participants were observed at more than one site. For over
566 hours of observation, study personnel used the tablet-based
WOMBAT app to record which information resources werebeing used, including the content types they accessed and the
proportion of time emergency department physicians spentaccessing the ANP during each 90-min observation. To evaluate
whether the method was correctly applied, the percentages oftime spent on high-level task categories were calculated andcompared for the 4 emergency departments (Figure 1). The
proportions of time spent on the different categories werecompared and found to be statistically similar across the sites.
The mean proportions of time spent (with 95% CIs) on task
categories in this study were compared with previously
published literature (Figure 1). These percentages of time spent
on different tasks while physician participants provided
emergency department care in Australian hospital wards weresimilar to the values reported in this study than the percentagesof time spent on the high-level task categories by physicians
providing care in Australian hospital wards [20,23].
The median percentage of physician participant time spent using
ANP was 8.5% at UAH (Figure 2; interquartile range, IQR,
3.7%-13.3%), 4.4% at GNCH (IQR 2.4%-4.4%), 4.6% at FMC(IQR 2.4%-7.6%), and 5.1% at PLC (IQR 3.0%-7.7%). The
value at UAH was significantly higher than the other sites(P<.001). The tasks physicians were observed completing mostoften while using ANP are identified in Figure 2.
JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 1. Comparison of proportions of time spent on different tasks in the 4 emergency departments (A) and 2 other studies using the Work Observation
Method by Activity Timing (WOMBAT) method (B). (A) The median and interquartile range of the data for individual observations of physician workat 4 EDs are shown by the symbol and the gray bars. The different task categories are labeled on the x-axis at the top of the figure. University of Alberta
Hospital: filled squares; Grey Nuns Community Hospital: filled circles; Foothill Medical Centre: empty squares; Peter Lougheed Centre: empty circles.
(B) The mean time spent and 95% CIs are reported for this study (+ sign), an Australian study of emergency department physician work (at a registrar
training level comparable with the participants in the current work; x sign), and an Australian study of hospital ward physician work (* sign).

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 2. Percentage of observed participant time spent using Alberta Netcare Portal (ANP). (A) Each circle represents the percentage of time spentusing ANP for a single 90-min observation session (n=376). The middle black bars represent the median percentage of time spent using ANP and thegray rectangles represent the interquartile range. University of Alberta Hospital site is A, Grey Nuns Community Hospital is B, Foothill Medical Centre
is C, and Peter Lougheed Centre is D. (B) Circles represent the proportion of time spent during individual observation sessions using ANP while
completing the tasks named at the left-hand side.

ANP was used during 20.9% of time spent on indirect patientcare, including reviewing patient information. ANP was usedalongside discussions. Of the total recorded time, 1.5% of therecorded time was spent on Professional Communication. Other
tasks that were completed while physicians used ANP included
Documentation (0.59%), Direct Patient Care (0.55%),
Supervision/Education (0.26%), In Transit (0.06%), andMedication-related tasks (0.01%).
To evaluate other factors that may affect the observed use of
ANP, physician demographic and practice characteristics wererecorded and simple linear regression was used to identify
potential effects on ANP use (Table 2). No statisticallysignificant effects were found.

Administrative Data Matching and Review
Audit logs from the same time period as the structured clinicalobservations (October 2014 to March 31, 2016) were comparedwith the triangulated findings between the 2 methods ofevaluating clinician use. Alongside the structured clinicalobservations, administrative access audit data from ANP werelinked with a routinely generated dataset on emergency
department visits. Patient visits to the 4 study emergency
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

departments between October 2014 and April 2016 wereidentified using administrative data and matched with accessaudit data to derive the number of ANP screens accessed during
3 periods of time (October 2014 through January 2015, July
2015 to September 2015, and February 2016 to April 2016).
An ANP access relating to an emergency department visit was
defined as the use of the ANP after presentation to theemergency department on the same and next calendar day.
During the study period, emergency department physicians at
all sites accessed 1,994,334 lab results, 763,334 diagnostic
imaging results, and 666,222 textual reports. Over 376
observations and 142 emergency department physicians, themean and median number of ANP screens viewed per 90-min
observation in the Edmonton sites were 53 (UAH) and 27(GNCH). In the Calgary sites, the mean number of ANP screensviewed was 15 (FMC) and 13 (PLC; Figure 3). Screen accesses
were calculated at each site, and a median of 20 screens wasaccessed per patient visit at UAH (IQR 6-67), 9 at GNCH (IQR
4-29), 7 at FMC (IQR 2-18), and 5 at PLC (IQR 2-14). When
compared with the structured clinical observations, the statisticalanalysis of screen access data also showed that ANP was used
more at UAH than the other sites.
JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Among the different content types available in ANP, laboratory
and imaging data were accessed more often than transcribedreports, dispensed drug information, or information notcategorized elsewhere (eg, electrocardiograms; Figure 4).

Graham et al
Physicians were observed spending relatively more timereviewing textual reports (mean 4.6%, 95% CI 4.1-5.1) thanlab results (mean 2.6%, 95% CI 2.1-3.1) or diagnostic imaging(mean 1.9%, 95% CI 1.6-2.1; Figure 4).

Table 2. Effect of demographic and practice characteristics on Alberta Netcare Portal (ANP) use as measured by the proportion of time spent.
Demographic and practice characteristics

Sites A and B (n)Sites C and D (n)Age (in years)

.68

<30

2

3

—a

30-39

24

38

—

40-49

23

25

—

50-59

7

10

—

>60

6

4

—

Sex

.06
Female

13

17

—

Male

49

63

—

Credentials.06

Medical Doctorate

41

78

—

Fellow of the Royal College of Physicians of Canada–Emergency
Medicine

27

44

—

American Board of Emergency Medicine

4

5

—

Special Competence in Emergency Medicine

28

29

—

Other (PhD)

3

0

—

Payment schemeaRelationship with observed
ANP usage (P value).47

Fee for service

54

73

—

Salary79—Mixed-compensation model

4

0

—

Years in practice, mean (SD)

13.3 (10)12.9 (9).29

Comfort with computersb, mean (SD)

8.1 (1.6)

8.3 (1.6).35

Comfort with electronic medical recordsb, mean (SD)

7.9 (1.6)

8.1 (1.7).93

Not applicable.

b

Likert scale 0-10; 10=completely comfortable.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 3. Relative usage of Alberta Netcare Portal across 4 emergency departments from 2014 to 2016 based on access audit data. Each symbol
represents the median weekly count of screens accessed per patient visit to the 4 emergency departments. University of Alberta Hospital: filled squares;
Grey Nuns Community Hospital: filled circles; Foothill Medical Centre: empty squares; Peter Lougheed Centre: empty circles.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al

Figure 4. Usage of the different clinical functions in Alberta Netcare Portal based on (A) median counts of screen views from access audit data and(B) observed app usage as measured by the percentage of time spent. The data are shown for each of the 4 sites (A: University of Alberta Hospital; B:
Grey Nuns Community Hospital; C: Foothill Medical Centre; D: Peter Lougheed Centre) for laboratory results (Lab), diagnostic imaging (Imaging),
textual reports (Reports), Pharmaceutical Information Network (PIN; ie, dispensed medication information), and other information (Other). Circles in(B) represent values for individual 90-min observations. Horizontal black bars and gray areas in (B) represent the median and 95% CIs, respectively.

Discussion

resulting in inconsistent and poorly understood clinical impacts[28].

Principal Findings

The triangulation of results in our study, along with thecomparison of the time and motion data with results that havebeen previously reported using the same method in othercountries and care environments, supports the findings.
Clinicians spent a significant amount of time accessing thevarious areas of the ANP and reviewing information relevant
to patient care, consistent with their content. The access audit
data complement these findings. The counts of screen accesses
show information searching behaviors that are temporally linked
to individual patient emergency department visits and implying
they are of important clinical value. Physician perceptions of
the ANP were explored in more detail via semistructured
interviews and are reported elsewhere.

This work triangulates data to evaluate the use of an iEHR
(health information exchange-type) in 4 Canadian emergency
departments. To our knowledge, there have been no prior
Canadian studies describing the use and utility of an iEHR in
the emergency department or any other clinical setting. AcrossCanada, iEHRs are at various stages of implementation andmaturity and have evolved according to provincial and territorial
strategies and priorities [16]. The Alberta Netcare Portal iEHR
is widely used throughout Alberta. A previous study of ANP
use showed 76% of users indicated that it helped provide quality
patient care, whereas 82% felt that it integrated easily into theirclinical workflow [18]. Canadian survey data from 2006 to 2014
demonstrate that in 6 provinces, the Canadian iEHR is perceived
to have positive outcomes in terms of user satisfaction, impact
on quality of care, and impact on productivity [1,15].
In the emergency department setting in other jurisdictions, thesetypes of read-only iEHRs have been shown to be valued byphysicians [2,24,25] and to increase the ability to identify
frequent emergency department users [26]. Usage of iEHRs has
been reported to vary across different practice settings evenwhen the technical configuration is identical [27], potentially
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

In Alberta, it is clear that the ANP has become an integral partof emergency department care and is used extensively, with thehighest observed usage in the complex environment of apaper-based tertiary care academic center. ANP was moreheavily used at the UAH and GNCH (where they document
care primarily on paper and order laboratory testing, diagnostic
imaging, and other interventions either using paper forms or byverbally or telephonically communicating with different
services), compared with sites using a clinical information
JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
system that provided similar information but greater integration
into point-of-care workflows. In contrast to the UAH, the tertiarycare academic site in the Calgary Zone (FMC) had an emergency
department clinical information system, and their relative useof ANP was considerably less—most of the information related
to their local patients was usually embedded within the context
of the emergency department clinical information system.
EHR disuse may result when users and owners do not accrue
the benefits of their use [29]. Here, we found that ANP use wasindependent of the characteristics we evaluated and consistent
with emergency department physicians perceiving that content
available via ANP supported medical decision making. In
particular, information related to the medical management ofpatients and clinical decision support was not available in ANP
and was noted as a potential area of improvement by participantsin the structured clinical observations.
Given that the ANP was utilized less frequently in Calgary
emergency departments where they were able to access muchof the ANP information in their regional clinical information
system, as clinical information system implementations increase
across the country, the role of separate provincial iEHRs needs
further evaluation. Physicians also seek to have clinicaldocumentation, electronic referral, computerized provider order
entry, and clinical decision support all accessible in a single
point-of-care system to support their clinical practice, rather
than logging in and out of multiple systems. Currently, the iEHR
is a read-only app (with some small exceptions) and does notsupport the clinical documentation and ordering needs. Potential
important benefits of clinical information system and electronic
records relate to patient safety, particularly around medication
management and clinical decision support [2,5-7,18], which are
not available in the ANP.

Limitations
Although hybrid paper and computerized practice environments
are common in Canada, our results may or may not generalize
to other settings that do not have the particular mix of thesesystems present in the study emergency departments. The
Alberta context, where an iEHR is available alongside a moretransactional clinical information system in some sites, afforded

Graham et ala unique opportunity to study its use and clinical utility
systematically.
In defining the access of ANP related to an emergency
department visit, the audit logs were not detailed enough to
identify from what part of the hospital the ANP was accessed.
Our ANP accesses may be slight overestimates when a patient
was admitted to the hospital and the ANP was accessed by the
inpatient care team. We believe this possibility to be small
because the majority of patients presenting were not admitted
to an inpatient service, and it can take many hours for an
admitted patient to move from an emergency department to theinpatient bed.
More generally, partially adopted clinical documentation
applications may lead to hybrid environments where some forms
of information are charted on paper and others are charted in aclinical information system [30-32]. This work illuminates theneed to use multiple methods while evaluating the impacts ofdifferent methods of information storage and retrieval in thecontext of fast-paced emergency department care while showing
the clinical utility of a single iEHR accessible across a province.

Conclusions
The current evaluation shows that ANP iEHR is well utilized
at the 4 sites studied, and physicians participating in the study
perceived ANP has a positive impact on knowledge of theirpatients, patient safety, and quality and continuity of care.
Physicians at the paper-based tertiary care hospital utilized ANP
markedly more than those at the clinical information
system-based tertiary care hospital or the 2 community hospitalsites. Physicians working at all 4 sites accessed lab results anddiagnostic imaging more often than textual reports, such asdischarge summaries and operative reports, but spent relatively
more time reviewing textual reports. Physician demographic
and practice characteristics did not predict this common usage
pattern. In its current form, ANP features that could be enhanced
include electronic referral, clinical documentation, andmedication ordering and management. Given the trend ofmoving toward comprehensive clinical information systems to
run hospital systems, the future design, development, andimportance of the iEHR need further evaluation.

Acknowledgments
The authors would like to acknowledge funding support from Alberta Health and Canada Health Infoway. The authors would
also like to acknowledge support in kind from the operational support team for Alberta Netcare Portal and the Chief Medical
Information Office of Alberta Health Services and for permission to access audit log data by the Alberta Health Chief Information
Office. The authors would like to thank Lauren Hirsch and Joyce Tse for their contributions to data collection and Nadia Dow
from the University of Alberta Department of Pediatrics and Tiffany Junghans from the University of Calgary Department ofEmergency Medicine for their coordination of the study. The authors also thank all the emergency department physicians who
participated in the study.

Conflicts of Interest
None declared.

Multimedia Appendix 1
WOMBAT Workflow Definitions.
[PDF File (Adobe PDF File), 89KB-Multimedia Appendix 1]
http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et alMultimedia Appendix 2
Access audit data definitions.
[XLSX File (Microsoft Excel File), 9KB-Multimedia Appendix 2]

References
1.
2.

3.

4.
5.

6.

7.

8.
9.
10.
11.
12.

13.

14.

15.

16.
17.
18.
19.
20.

21.

Strating D, MacGregor J, Campbell N. Bridging information “islands”: designing an electronic health record that meetsclinical needs. Healthc Q 2006;9(2):94-98. [Medline: 16640139]
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information
technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006 May 16;144(10):742-752. [Medline:
16702590]
Longhurst CA, Parast L, Sandborg CI, Widen E, Sullivan J, Hahn JS, et al. Decrease in hospital-wide mortality rate after
implementation of a commercially sold computerized physician order entry system. Pediatrics 2010 Jul;126(1):14-21. [doi:10.1542/peds.2009-3271] [Medline: 20439590]
Goldzweig CL, Towfigh A, Maglione M, Shekelle PG. Costs and benefits of health information technology: new trends
from the literature. Health Aff (Millwood) 2009;28(2):w282-w293. [doi: 10.1377/hlthaff.28.2.w282] [Medline: 19174390]
Poon EG, Keohane CA, Yoon CS, Ditmore M, Bane A, Levtzion-Korach O, et al. Effect of bar-code technology on thesafety of medication administration. N Engl J Med 2010 May 6;362(18):1698-1707. [doi: 10.1056/NEJMsa0907115][Medline: 20445181]
Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient
outcomes: a multiple hospital study. Arch Intern Med 2009 Jan 26;169(2):108-114. [doi: 10.1001/archinternmed.2008.520][Medline: 19171805]
Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent
literature shows predominantly positive results. Health Aff (Millwood) 2011 Mar;30(3):464-471 [FREE Full text] [doi:10.1377/hlthaff.2011.0178] [Medline: 21383365]
Jones SS, Adams JL, Schneider EC, Ringel JS, McGlynn EA. Electronic health record adoption and quality improvement
in US hospitals. Am J Manag Care 2010 Dec;16(12 Suppl HIT):SP64-SP71 [FREE Full text] [Medline: 21314225]
Lau F, Kuziemsky C, Price M, Gardner J. A review on systematic reviews of health information system studies. J Am Med
Inform Assoc 2010 Nov;17(6):637-645 [FREE Full text] [doi: 10.1136/jamia.2010.004838] [Medline: 20962125]
Himssanalytics. Health Information Management Systems Society URL: http://www.himssanalytics.org/ [accessed2017-12-08] [WebCite Cache ID 6vYrR1QBJ]
Infocentral.infoway-inforoute. 2017. Standards Centre URL: https://infocentral.infoway-inforoute.ca/en/standards/
standards-centre [accessed 2017-12-08] [WebCite Cache ID 6vYpoEJNM]
Infoway-inforoute. 2016. Digital health blueprint URL: https://www.infoway-inforoute.ca/en/component/edocman/
2944-digital-health-blueprint-enabling-coordinated-collaborative-health-care/download?Itemid=0[WebCite Cache ID
6vYqGY3So]
Georgiou A, Prgomet M, Paoloni R, Creswick N, Hordern A, Walter S, et al. The effect of computerized provider order
entry systems on clinical care and work processes in emergency departments: a systematic review of the quantitative
literature. Ann Emerg Med 2013 Jun;61(6):644-53.e16. [doi: 10.1016/j.annemergmed.2013.01.028] [Medline: 23548404]
Hersh WR, Totten AM, Eden KB, Devine B, Gorman P, Kassakian SZ, et al. Outcomes from health information exchange:systematic review and future research needs. JMIR Med Inform 2015 Dec 15;3(4):e39 [FREE Full text] [doi:10.2196/medinform.5215] [Medline: 26678413]
Tharmalingam S, Hagens S, Zelmer J. The value of connected health information: perceptions of electronic health record
users in Canada. BMC Med Inform Decis Mak 2016 Jul 16;16:93 [FREE Full text] [doi: 10.1186/s12911-016-0330-3][Medline: 27422571]
Gheorghiu B, Hagens S. Measuring interoperable EHR adoption and maturity: a Canadian example. BMC Med Inform
Decis Mak 2016 Jan 25;16:8 [FREE Full text] [doi: 10.1186/s12911-016-0247-x] [Medline: 26810606]
Albertanetcare. The History of the EHR URL: http://albertanetcare.ca/history.htm [accessed 2017-12-08] [WebCite Cache
ID 6vYt72HAx]
Mayes DC, Mador DR. Evaluating the value and impact of an electronic health record in a complex health system.
ElectronicHealthcare 2010 Mar;8(4):e3-14.
Albertanetcare. EHR Usage Statistics URL: http://albertanetcare.ca/statistics.htm [accessed 2017-12-08] [WebCite Cache
ID 6vYsn6N9x]
Westbrook JI, Ampt A. Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT)
to measure clinicians' patterns of work and communication. Int J Med Inform 2009 Apr;78 Suppl 1:S25-S33. [doi:10.1016/j.ijmedinf.2008.09.003] [Medline: 18951838]
Ballermann MA, Shaw NT, Mayes DC, Gibney RT, Westbrook JI. Validation of the Work Observation Method By Activity
Timing (WOMBAT) method of conducting time-motion observations in critical care settings: an observational study. BMC
Med Inform Decis Mak 2011 May 17;11:32 [FREE Full text] [doi: 10.1186/1472-6947-11-32] [Medline: 21586166]

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 11(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
22.
23.
24.
25.

26.

27.

28.

29.
30.

31.
32.

Graham et al

Cihi. National Ambulatory Care Reporting System Metadata (NACRS) URL: https://www.cihi.ca/en/
national-ambulatory-care-reporting-system-metadata [accessed 2018-05-22] [WebCite Cache ID 6zcDYIkmr]
Westbrook JI, Ampt A, Kearney L, Rob MI. All in a day's work: an observational study to quantify how and with whom
doctors on hospital wards spend their time. Med J Aust 2008 May 5;188(9):506-509. [Medline: 18459920]
Westbrook JI, Coiera E, Dunsmuir WT, Brown BM, Kelk N, Paoloni R, et al. The impact of interruptions on clinical taskcompletion. Qual Saf Health Care 2010 Aug;19(4):284-289. [doi: 10.1136/qshc.2009.039255] [Medline: 20463369]
Johnson KB, Unertl KM, Chen Q, Lorenzi NM, Nian H, Bailey J, et al. Health information exchange usage in emergency
departments and clinics: the who, what, and why. J Am Med Inform Assoc 2011 Sep;18(5):690-697 [FREE Full text] [doi:10.1136/amiajnl-2011-000308] [Medline: 21846788]
Shapiro JS, Johnson SA, Angiollilo J, Fleischman W, Onyile A, Kuperman G. Health information exchange improves
identification of frequent emergency department users. Health Aff (Millwood) 2013 Dec;32(12):2193-2198. [doi:10.1377/hlthaff.2013.0167] [Medline: 24301405]
Lanham HJ, Leykum LK, McDaniel Jr RR. Same organization, same electronic health records (EHRs) system, different
use: exploring the linkage between practice member communication patterns and EHR use patterns in an ambulatory caresetting. J Am Med Inform Assoc 2012;19(3):382-391 [FREE Full text] [doi: 10.1136/amiajnl-2011-000263] [Medline:
21846780]
Healthit.ahrq. 2011. Guide to Reducing Unintended Consequences of Electronic Health Records URL: https://healthit.
ahrq.gov/sites/default/files/docs/publication/guide-to-reducing-unintended-consequences-of-electronic-health-records.
pdf[WebCite Cache ID 6vYsn6N9x]
Sittig DF, Singh H. Legal, ethical, and financial dilemmas in electronic health record adoption and use. Pediatrics 2011
Apr;127(4):e1042-e1047 [FREE Full text] [doi: 10.1542/peds.2010-2184] [Medline: 21422090]
Casalino LP, Dunham D, Chin MH, Bielang R, Kistner EO, Karrison TG, et al. Frequency of failure to inform patients ofclinically significant outpatient test results. Arch Intern Med 2009 Jun 22;169(12):1123-1129. [doi:10.1001/archinternmed.2009.130] [Medline: 19546413]
Callen JL, Westbrook JI, Braithwaite J. The effect of physicians' long-term use of CPOE on their test management work
practices. J Am Med Inform Assoc 2006;13(6):643-652 [FREE Full text] [doi: 10.1197/jamia.M2152] [Medline: 16929035]
Stausberg J, Koch D, Ingenerf J, Betzler M. Comparing paper-based with electronic patient records: lessons learned during
a study on diagnosis and procedure codes. J Am Med Inform Assoc 2003;10(5):470-477 [FREE Full text] [doi:10.1197/jamia.M1290] [Medline: 12807808]

Abbreviations
AHS: Alberta Health Services
ANP: Alberta Netcare Portal
EHR: electronic health record
FMC: Foothill Medical Centre
GNCH: Grey Nuns Community HospitaliEHR: interoperable electronic health record
IQR: interquartile range
NACRS: National Ambulatory Care Reporting System
PLC: Peter Lougheed Centre
UAH: University of Alberta Hospital
WOMBAT: Work Observation Method by Activity Timing

Edited by G Eysenbach; submitted 20.02.18; peer-reviewed by F Lau, B Gheorghiu; comments to author 29.03.18; revised version
received 23.05.18; accepted 24.05.18; published 25.09.18
Please cite as:
Graham TAD, Ballermann M, Lang E, Bullard MJ, Parsons D, Mercuur G, San Agustin P, Ali S
Emergency Physician Use of the Alberta Netcare Portal, a Province-Wide Interoperable Electronic Health Record: Multi-Method
Observational StudyJMIR Med Inform 2018;6(3):e10184
URL: http://medinform.jmir.org/2018/3/e10184/
doi: 10.2196/10184
PMID: 30274967

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 12(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Graham et al©Timothy AD Graham, Mark Ballermann, Eddy Lang, Michael J Bullard, Denise Parsons, Gabriella Mercuur, Pat San Agustin,
Samina Ali. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 25.09.2018. This is an open-accessarticle distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR
Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on
http://medinform.jmir.org/, as well as this copyright and license information must be included.

http://medinform.jmir.org/2018/3/e10184/

XSL• FO
RenderX

JMIR Med Inform 2018 | vol. 6 | iss. 3 | e10184 | p. 13(page number not for citation purposes)