JMIR MEDICAL INFORMATICS

Wurster et al

Original Paper

The Implementation of an Electronic Medical Record in a German
Hospital and the Change in Completeness of Documentation:
Longitudinal Document Analysis
Florian Wurster1, MSc; Marina Beckmann1, DPhil; Natalia Cecon-Stabel1, MSc; Kerstin Dittmer1, MA; Till Jes
Hansen1, MSc; Julia Jaschke2, MSc; Juliane Köberlein-Neu2, Prof Dr; Mi-Ran Okumu1, MA; Carsten Rusniok1, MA;
Holger Pfaff1, Prof Dr; Ute Karbach1, PD, Dr
1

Chair of Quality Development and Evaluation in Rehabilitation, Institute of Medical Sociology, Health Services Research, and Rehabilitation Science,
Faculty of Human Sciences & Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
2

Center for Health Economics and Health Services Research, University of Wuppertal, Wuppertal, Germany

Corresponding Author:
Florian Wurster, MSc
Chair of Quality Development and Evaluation in Rehabilitation, Institute of Medical Sociology, Health Services Research, andRehabilitation Science, Faculty of Human Sciences & Faculty of Medicine and University Hospital Cologne, University of
CologneEupener Str. 129
Cologne, 50933
Germany
Phone: 49 22147897116
Email: florian.wurster@uni-koeln.de

Abstract
Background: Electronic medical records (EMR) are considered a key component of the health care system’s digital transformation.
The implementation of an EMR promises various improvements, for example, in the availability of information, coordination ofcare, or patient safety, and is required for big data analytics. To ensure those possibilities, the included documentation must beof high quality. In this matter, the most frequently described dimension of data quality is the completeness of documentation. In
this regard, little is known about how and why the completeness of documentation might change after the implementation of an
EMR.
Objective: This study aims to compare the completeness of documentation in paper-based medical records and EMRs and to
discuss the possible impact of an EMR on the completeness of documentation.
Methods: A retrospective document analysis was conducted, comparing the completeness of paper-based medical records and
EMRs. Data were collected before and after the implementation of an EMR on an orthopaedical ward in a German academic
teaching hospital. The anonymized records represent all treated patients for a 3-week period each. Unpaired, 2-tailed t tests,
chi-square tests, and relative risks were calculated to analyze and compare the mean completeness of the 2 record types in generaland of 10 specific items in detail (blood pressure, body temperature, diagnosis, diet, excretions, height, pain, pulse, reanimation
status, and weight). For this purpose, each of the 10 items received a dichotomous score of 1 if it was documented on the firstday of patient care on the ward; otherwise, it was scored as 0.
Results: The analysis consisted of 180 medical records. The average completeness was 6.25 (SD 2.15) out of 10 in the paper-based
medical record, significantly rising to an average of 7.13 (SD 2.01) in the EMR (t178=–2.469; P=.01; d=–0.428). When looking
at the significant changes of the 10 items in detail, the documentation of diet (P<.001), height (P<.001), and weight (P<.001)
was more complete in the EMR, while the documentation of diagnosis (P<.001), excretions (P=.02), and pain (P=.008) was lesscomplete in the EMR. The completeness remained unchanged for the documentation of pulse (P=.28), blood pressure (P=.47),
body temperature (P=.497), and reanimation status (P=.73).
Conclusions: Implementing EMRs can influence the completeness of documentation, with a possible change in both increased
and decreased completeness. However, the mechanisms that determine those changes are often neglected. There are mechanismsthat might facilitate an improved completeness of documentation and could decrease or increase the staff’s burden caused byhttps://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wurster et al

documentation tasks. Research is needed to take advantage of these mechanisms and use them for mutual profit in the interestsof all stakeholders.
Trial Registration: German Clinical Trials Register DRKS00023343; https://drks.de/search/de/trial/DRKS00023343(JMIR Med Inform 2024;12:e47761) doi: 10.2196/47761
KEYWORDS
clinical documentation; digital transformation; document analysis; electronic medical record; EMR; Germany; health services
research; hospital; implementation

Introduction
The digital transformation of the health care system isconsidered an essential subject to meet current and futuresocietal challenges such as an aging population or rising healthcare expenditures while at the same time maintaining a highquality of care [1]. An important early step in hospitals’
digitalization and a fundamental requirement for expanding
digital maturity is the implementation of an electronic medical
record (EMR) [2]. This EMR is considered to be an “electronic
record of health care information of an individual that is created,gathered, managed, and consulted by authorized clinicians andstaff within 1 health care organization” [3] and replaces theinternal clinical documentation on preprinted paper-based charts.
Studies show that the implementation of an EMR can lead tovarious improvements in the clinical context (eg, in theavailability of information [4], coordination of care [5], or
patient safety [6]). Moreover, the EMR facilitates the secondary
usage of the documented data for research purposes through its
digital accessibility [7]. To reach those benefits, it isindispensable that the EMR contain documentation that is ofhigh quality. However, there are varying definitions regarding
the quality of documentation. In that matter, the Institute ofMedicine defined completeness, legibility, accuracy, andmeaning as the main aspects of a medical record’s data quality[8]. For those, the completeness of documentation was shown
to be the most common dimension of data quality whenempirically analyzing the documentation in EMRs [9], and it
was highlighted to be especially important for secondary uses
such as big data analyses [10].
Our recent systematic review also stated the completeness of
documentation as the state of the art for the comparison of
paper-based and EMRs [11]. This comparison is important since
the implementation of an EMR and the associated transition
from handwritten documentation to digital documentation canheavily affect the documentation subject since the transition
offers the possibility to adjust which information has to be
documented in which way [12]. For example, digitization
enables the adoption of certain functionalities that can alter thecompleteness of documentation, like automatically transferring
information from other digital devices to the EMR [13].
Moreover, when working with the EMR, information can be
documented remotely, while the paper-based medical record
had to be located and physically accessed first. In this matter,several studies conducted in the inpatient setting showed
increased completeness in the EMR compared to the paper-based
medical record, for example, for the documentation of signs
and symptoms [13,14], weight and height, or malnutrition
https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

screening [15]. This suggests that the implementation of an
EMR might lead to improvements in the completeness of
documentation in general. It is therefore the main purpose ofthis study to evaluate the change in completeness due to theimplementation of an EMR in an inpatient setting. Literature
already provides proof of a change of completeness in regard
to some specific documented information that is analyzed inthis work (eg, the documentation of vital signs) [13,14]. Those
empirical results might thus be validated for the presented
work’s specific setting and discipline. In addition, some of theinformation that is analyzed in this work is not described inliterature yet (eg, the documentation of pain). It is examined
for the first time with regard to changes in completeness after
the implementation of an EMR.
The knowledge gained can not only support the implementationof new EMRs but could also help understand and optimize
arising changes in documentation when existing EMRs need tobe adapted [16,17]. This is an important aspect, as theimplementation of new EMRs is described as one of the mostimportant interventions to improve the quality of documentation[18]. In this process, mechanisms affecting the completenessof documentation in medical records are not completely
understood [10]. On the other hand, this knowledge is needed
to fulfill reported educational needs regarding how to reach the
optimum quality of documentation [19]. In this context, thisstudy contributes to a more comprehensive understanding of
the impact of an EMR on the quality of documentation.

Methods
Overview
This study follows the “Strengthening the Reporting ofObservational Studies in Epidemiology” (STROBE) statement[20] whenever it is applicable. It offers reporting standards toensure the reporting of any important information in empiricalresearch studies. A checklist with details, where the STROBE
information is mentioned in the manuscript, can be found in
Multimedia Appendix 1.

Ethical Considerations
The study has been approved by the ethics committee of theMedical Faculty of the University of Cologne, Germany(20-1349). All data was anonymized at all times during thescientific analysis. No compensation was paid.

Setting and Participants
The study took place as part of the research project eCoCo,
which Beckmann et al [21] described in detail. Within theeCoCo project, the researchers collected various types of data
JMIR Med Inform 2024 | vol. 12 | e47761 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS(observations, surveys, interviews, documents, andadministrative data) to investigate a possible change ininterprofessional collaboration and clinical workflows following
the implementation of an inpatient EMR. This study is part of
the related work package on documentation content and quality,
which took place in a large academic teaching hospital in
Germany. The hospital replaces its internal documentation on
preprinted paper-based charts with a commercial EMR system(Meona; Mesalvo Freiburg GmbH). The EMR runs on multiple
computers that can be moved flexibly over the ward on trolleys.
The study follows a pre-post design, retrospectively analyzing
the content of the medical records before and after the
implementation of the EMR on the hospitals’ orthopaedical
ward. Within the first measuring phase, the paper-based medical
records were provided as a digital copy of the paper sheets.
Those paper-based records represent all patients who weretreated on the ward during the last 3 weeks in November 2020
(t0). After 6 months, employees received training on how to
use the EMR before the implementation of the EMR took placein May 2021. The EMRs were again provided as a digital copy
within a second measuring phase, representing all patients who

Wurster et al
were treated on the same ward during the first 3 weeks of August2022 (t1). This resulted in a gap of 15 months between the firstand second measuring phases. The complete data set wasavailable to the research team in November 2022 (Figure 1).
The hospital provided anonymized medical records to theresearch team after the records were archived and cleared ofsensitive personal data (eg, the patient’s name or date of birth)in the hospital’s internal processes. Any assignment of thepatient data or linking of the records’ contents to any individual
patient was therefore impossible for the research team, whichis, thus, in compliance with the European Union General Data
Protection Regulation. This also implies the absence ofsociodemographic information for describing the comparedsamples. The hospital’s mandatory annual quality report, whichis available to the public through a designated database [22], istherefore used to describe the ward’s patient sample and theperformed treatments in general. This allows an approximation
to a description and comparison of the compared samples interms of their International Classification of Diseases(ICD)–diagnoses distribution.

Figure 1. Data collection. EMR: electronic medical record.

Study Objective
To answer the question of a possible change in completeness,the records were analyzed by content [23]. The change frompaper-based documentation to EMRs always offers theopportunity to fundamentally change the structure of the records.
This was shown exemplarily by Montagna et al [12] when the
documentation as a continuously written text in the paper-based
record was changed to a list of events in the EMR. It is therefore

https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

important to ensure the comparability between the 2 record
types for the purpose of analyzing a possible change in theircompleteness. To achieve comparability, the medical records
progress note was selected as a specific object of interest forthis study’s analyses since it retained the same structure andformat in both record types. Part of this progress note is thefever chart (Figures 2 and 3), which includes basic details about
vital signs, personal health data, etc [24].

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wurster et al

Figure 2. Paper-based fever chart.

Figure 3. Electronic fever chart.

All information that was commonly documented in both of the
2 record types (paper-based and electronic) became part of thiswork. Weiskopf and Weng [9] described this selection
mechanism for assessing data quality based on the parallels
https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

between the EMR and the paper-based record. This procedure
resulted in a total of 10 key items that were analyzed forcompleteness in this work: blood pressure, body temperature,
diagnosis, diet, excretions, height, pain, pulse, reanimation
JMIR Med Inform 2024 | vol. 12 | e47761 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
status, and weight. The documentation of this information isequally possible and performed by nurses and physicians.
However, there is no information available about who
specifically entered the information.
All of those items should be documented immediately when
patient care begins on the ward [25]. However, while thedocumentation of vital signs can take place up to several times
a day, the documentation of the patient’s diet usually occurs
once a day, and the documentation of the reanimation status(patient’s preference regarding a possible resuscitation) is
probably documented only once per hospital stay. Because of
these varying documentation practices and to ensurecomparability, the analysis focuses on certain documentationin the progress notes that was entered on the first day of patientcare on the ward. With regard to the documentation of adiagnosis, it is therefore the diagnosis with which a patient isadmitted to the hospital. This diagnosis is mainly responsible
for the allocation to specific medical specialties as well as acertain ward and does not necessarily have to match the final
diagnosis at the time of discharge, which is important for
reimbursement purposes.

Statistical Analysis of CompletenessFor every record, each of the 10 items received a dichotomousscore of 1 if it was documented on the first day of patient careon the ward; otherwise, it was scored as 0. This resulted in apercentage of completeness for each item per record type.
Chi-square tests for independence were used to assess
statistically significant differences in the percentage ofcompleteness per item between the 2 record types. Relative
risks were calculated for the association between the electronic
record type and a possible increase in completeness. To improve
the reliability of the associated confidence intervals, they were
calculated with 5000 bootstrap replications since the original
sample sizes are unbalanced. Moreover, the overall completenesswas assessed as sum of the 10 items, resulting in a mean scoreof completeness per record type ranging from 0 (no item

https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

Wurster et al
documented) to 10 (all 10 items documented). Those meanscores of completeness per record type were analyzed forequality of variance and statistical difference using unpaired,
2-tailed t tests. Assumptions were checked using several
methods (normal distribution: QQ plots and Shapiro-Wilk test;
homogeneity of variances: Levene test; and linearity: scatterplot). The level of significance was set to be P<.05 for all
calculations. The data were stored in Microsoft Excel (Microsoft
Corp) and analyzed in December 2022 using SPSS software(version 29; IBM Corp).

Results
Participants
During the first measuring phase (November 2020), a total of
44 patients (paper-based) were treated on the orthopaedical
ward. They were encountering a total of 136 treated patients(electronic) during the second measuring phase (August 2022).
This resulted in a total of 180 medical records that became partof this analysis. Due to the data protection regulation and theaccompanied anonymization of the records data, there is no
information regarding the demographics of the specific study
population. Therefore, the ward’s ICD-diagnosis distribution
is given as an approximation of a sample description. In 2020,the 3 most frequently coded diagnoses for the orthopaedical
ward were complications of internal orthopedic prosthetic
devices, implants and grafts (ICD-T84), dorsalgia (ICD-M54),
and fracture of shoulder and upper arm (ICD-S42). This report
is not yet published for 2022, but the top 3 treated diagnoses in2019 or 2021 were similar to those in 2020 (Table 1). It cantherefore be expected that the treated diagnoses will be similarin 2022, too. Another supporting fact is that the most frequently
performed procedure (surgical access to the lumbar spine, thesacrum, or the coccyx [coded as OPS-5-032 in the German
adaptation of the International Classification of Procedures in
Medicine which is part of the coding system for hospitals
reimbursement]) was the same in all 3 years (2019-2021).

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wurster et al

Table 1. Most frequently coded diagnoses.
ICDa Code

Values, nb/Nc (%)

2019
Dorsalgia (ICD-M54)

213/3147 (6.77)

Other spondylopathies (ICD-M48)

133/3147 (4.23)

Fracture of forearm (ICD-S52)

131/3147 (4.16)

2020
Complications of internal orthopedic prosthetic devices, implants and grafts (ICD-T84)

166/2912 (5.7)

Dorsalgia (ICD-M54)

148/2912 (5.08)

Fracture of shoulder and upper arm (ICD-S42)

121/2912 (4.16)

2021

a

Complications of internal orthopedic prosthetic devices, implants and grafts (ICD-T84)

164/3091 (5.3)

Dorsalgia (ICD-M54)

163/3091 (5.27)

Fracture of forearm (ICD-S52)

159/3091 (5.14)

ICD: International Classification of Diseases.

bFrequency of coded diagnosis.

c

Total inpatient cases.

Change of Completeness
The mean number of documented items was 6.25 (SD 2.15) out
of 10 in paper-based medical records and 7.13 (SD 2.01) out of
10 in EMRs. The Levene test confirmed the homogeneity ofvariances. The Shapiro-Wilk test did not confirm normal
distributions, but the QQ plots show an approximation to anormal distribution and a comparable degree of normality(Multimedia Appendix 2). The unpaired t test confirmed the
EMRs were statistically significantly more complete than thepaper-based medical records under equal variances in the 2record types (t178=–2.469; P=.01; d=–0.428). When looking at
the 10 items separately, data from chi-square tests showed thatthe documentation of diet increased from being present in 30%(13/44) of the paper-based medical record to 75% (102/136;

P<.001) in the EMR, height from 27% (12/44) to 85.3%(116/136; P<.001), and weight from 27% (12/44) to 86%(117/136; P<.001). At the same time, documentation ofdiagnosis decreased from being present in 100% (44/44) of thepaper-based medical records to 49% (66/136; P<.001) in the
EMR, excretions from 86% (38/44) to 68% (92/136; P=.02),
and pain from 95% (42/44) to 78% (106/136; P=.008). The
documentation of vital signs such as blood pressure (P=.47),
body temperature (P=.497), and pulse (P=.28) remained
unchanged on a high level of completeness, while thedocumentation of reanimation status (P=.73) remained
unchanged on a low level of completeness (Table 2). Positive
relative risks (Figure 4) illustrate the association of the electronic
record type (exposure) with complete documentation (outcome).
The confidence intervals represent 5000 bootstrap replications.

Table 2. Change of completeness.
VariableaType of record

Chi-square (df)

P valueRRa (95% CI)

Paper (n=44), n (%)

Electronic (n=136), n (%)

Blood pressure

37 (84)120 (88.2)

0.5 (1).47

1.05 (0.92-1.23)

Body temperature

36 (81.8)

117 (86)

0.5 (1).497

1.05 (0.91-1.25)

Diagnosis

44 (100)

66 (48.5)

37.1 (1)<.001

0.48 (0.40-0.57)

Diet

13 (29.6)

102 (75)

29.8 (1)<.001

2.54 (1.70-4.69)

Excretions

38 (86.4)

92 (67.7)

5.8 (1).02

0.78 (0.66-0.93)

Height12 (27.3)

116 (85.3)

54.5 (1)<.001

3.13 (2.06-5.91)

Pain

42 (95.4)

106 (77.9)

7.0 (1).008

0.82 (0.73-0.91)

Pulse

36 (81.8)120 (88.2)

1.2 (1).28

1.08 (0.94-1.29)

Reanimation status

5 (11.4)

13 (9.6)

0.1 (1).73

0.84 (0.33-3.33)

Weight12 (27.3)

117 (86)

56.5 (1)<.001

3.15 (2.07-6.00)

RR: relative risk.

https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wurster et al

Figure 4. Forest plot of relative risks.

Discussion
Principal Findings and Comparison to Previous Work
The main findings of this study confirm an improved
completeness of the analyzed information in the EMR on
average. This provides further evidence for the suggestion thatthe general completeness of documentation can improve after
the implementation of an EMR. The findings align with theresults of similar studies, showing improvements in other data
quality dimensions like the accuracy [26] or legibility [27] of
documentation. However, when looking at the completeness of
the analyzed 10 items in detail, the improvements can only beseen in 3 out of 10 items (diet, height, and weight), while 3
different items exhibited a deterioration in completeness(diagnosis, excretions, and pain). This links to the results ofCoffey et al [28], who found 5 of their 11 analyzed items to be
more complete while also proving 1 of their elements to be lesscomplete. The reason for the variation in the change incompleteness may lie in the mechanism of how information
reaches the record. In the paper-based medical records, allinformation was documented by hand by the various
professional groups. EMRs, on the other hand, offer technicalfeatures, for example, automatically obtaining information fromother digital sources, like patients’ health insurance data [29].
This was manifested as a possible mechanism by Jang et al [30],who showed improved completeness in the EMR for the
automatically filled information but not for the manually
documented ones.
The analysis shows that roughly every second EMR was missing
the documentation of a diagnosis. This is a remarkable change,
as it was present in every paper-based record (44/44, 100% vs
66/136, 48.5%). In the first place, it must be clarified that thediagnosis is determined by a physician who enters it into anindependently run hospital information system (HIS). This
documented diagnosis can also be a preliminary diagnosis,
which is used for distribution to the clinical disciplines and ispresent for every admitted patient. The HIS was already inoperation when medical staff was still using the paper-based
preprints for documentation purposes. After the EMR’s
https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

implementation, the HIS was still in operation along with the
EMR. That being said, it is undisputed that during thepaper-based period as well as the electronic period, a diagnosis
was indeed present for the patients. In the paper-based period,
the diagnosis was transferred manually from the HIS into thepaper-based preprints, when a record for a recently admitted
patient was prepared by a nurse. Since the HIS and the EMR
are produced by different software developers, the diagnosis
cannot be transferred automatically from the HIS into the EMR.
Due to this noninteroperability of the 2 independent digitalsystems, the manual transfer is still necessary in the electronic
period. With the drop of completeness in mind, this double
documentation was accepted and carried out in the period of
the paper-based record. In the electronic period, the describeddouble documentation has decreased. One possible explanationis that the HIS was not automatically accessible, when an
employee had the paper-based record at hand. With theintroduction of the EMR, the availability of the EMR became
synonymous with the availability of the HIS, since both areaccessible from a computer. Therefore, the transfer of thediagnosis from the HIS to the EMR may no longer have been
considered necessary. Nevertheless, the reason for this difference
remaining unclear illustrates that the sole analyzation ofcompleteness of the documentation alone does not provide
sufficient information about the actual quality of the provided
treatment. In that matter, it must also be highlighted that therecord can contain additional qualitative data entries, like freetexts, which might complement the analyzed quantitative
information. This underlines that an insufficient quality of
documentation does not necessarily allow conclusions to be
drawn about the quality of care, and vice versa.
Brown [31] emphasizes this by cautioning people to always
consider the circumstances under which people put information
into the record before drawing conclusions. This is a major issue
because the completeness of documentation might be biased
due to aspects that do not directly derive from clinical care. On
the one hand, the hospital’s reimbursement for the delivered
care depends on what is documented and might cause a possible
strengthened thorough filling of certain fields [32]. On the other
hand, the burden caused by documentation tasks is critically
JMIR Med Inform 2024 | vol. 12 | e47761 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
heavy. It is responsible for a high prevalence of burnout among
physicians and nurses [33]. Therefore, clinically or legally
unnecessary documentation might be evaded [34]. However,even though complete documentation might neither necessarily
arise from nor be essential for the delivery of excellent clinicalcare, it is likewise of concern under the aspect of big data
analytics. In this regard, it would be desirable for the discussed
diagnosis to indeed be present in the EMR, even if it alreadyexists in the HIS. An automatic transfer of this information
could help to prevent the burden on staff resulting from manualtransmission and ensure a complete data set. This is an important
point, as the insights gained from analyzing big data offer
numerous opportunities, like data-based personalized care in
diagnostics and therapy or the support of scientific activities,
both with the chance of saving lives and reducing health carecosts at the same time [7,35]. It is therefore indispensable torecognize the possibility of changes in documentation due tothe implementation or adaptation of EMRs. Only with thisattention will it become possible to optimize the documentation
process with a focus on the various benefits for all stakeholders,like patients [6], practitioners [36], organizations [5], and society[7].

Strengths and Limitations
The German health care system, in which the study was
conducted, was heavily strained by the high number of
COVID-19 cases and the associated use of intensive care units
during the study period. Especially the first measuring phase(November 2020) fell into the first pandemic year when manyplanned procedures were suspended to increase hospital
capacities. For the first lockdown period in Germany (March
2020), a decrease in orthopedic surgeries is described byapproximately 80% [37]. A lockdown-like situation was again
declared during the first measuring phase [38], which probably
explains the difference in treated patients over the 2 measuringphases (nPaper=44 vs nElectronic=136). However, the similarity
between the coded ICD diagnoses over different years (Table1) suggests that the proven changes in completeness of
documentation are not due to significant changes in the studied
patient sample, but a detailed sample description based on
socioeconomical data is missing due to data protection
regulations. On the other hand, there is a study assuming a
positive influence of the pandemic on the completeness of
documentation since an incomplete documentation might have
led to repetitive contacts with the patient, which could havebeen avoided if the documentation would have been completein the first place [39]. However, this cannot be verified in thispaper due to the lack of further measuring phases. Within thisgiven context, the generalizability of the presented results
remains limited.
Further, limitations regarding the analyzed data set have to bestated. The chosen unpaired t test is theoretically based on theassumption of normal distributions. This could not be confirmed

Wurster et al
statistically for the mean completeness scores by the
Shapiro-Wilk test. Although t test has been shown to be robust
to a missing normal distribution [40] and the QQ plots(Multimedia Appendix 2) indicate an approximation to a normal
distribution, the results could still be biased by the broken
assumption.
Moreover, the analyzed data set is missing any information onwhich person was entering the documentation regarding which
patient. On the one hand, it might be arguable that the same
physicians or nurses were documenting during the first and also
the second measuring phases. This circumstance would makethe 2 compared measuring phases dependent samples, having
an impact on the chosen statistical model. Since the analyzed
data set is missing this information, the results might be biased
regarding a possible dependent or independent sample. However,
the time passed between the 2 measuring phases might have led
to a change of the employees since the teaching status of thehospital results in many young physicians or nurses who do notnecessarily stay on the same ward for a long time. Moreover,the hospital in which the study was conducted has a rotation
system in which clinicians rotate hospital-wide across different
wards of the same discipline. Those 2 facts let us assume thatthe 2 compared samples are indeed independent. However, thelack of information regarding the documenting individual ispreventing the use of advanced tests like mixed effect models.
These could equally consider the record type on the one hand
and the possible documenting individuals on the other hand,potentially advancing the results’ reliability. However, the15-month interval from the implementation date of the EMR
to the second data collection signifies that there is only little
risk of any possible changes in documentation due to a bias
from the described effects of preimplementational
documentation training [41] since the employees indeed
underwent software training before they were allowed to use
the EMR. Therefore, the shown changes in completeness are,in fact, most likely due to the implementation of the EMR.

Conclusions
The results show that implementing EMRs can influence thecompleteness of documentation. A demonstrated improved
completeness might also facilitate an improvement of thedescribed outcomes that depend on documentation that is ofhigh quality, like the availability [4] and analyzability of
information [7,35], the coordination of care [5], or patient safety[6]. However, at the same time, the results show that adeterioration of completeness is also conceivable with theaccompanied risks. This highlights the importance ofunderstanding the underlying mechanisms that determine thesechanges. The knowledge may help stakeholders manage theimplementation of new EMRs or the optimization of existing
EMRs. Future research should address mechanisms that can
improve documentation while simultaneously reducing theburden on practitioners caused by documentation tasks.

Acknowledgments
This work is funded by the German Federal Ministry of Education and Research (grant 01GP1906B). The sponsor had no influenceon study design, data collection, analysis, or the writing process.
https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Wurster et al

Data Availability
The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions
FW and UK conceptualized the article. Data collection and analysis were performed by FW, supervised by UK. The originaldraft of the manuscript was written by FW, and all authors reviewed and edited previous versions of the manuscript and contributed
to the interpretation of the data. All authors read and approved the final manuscript.

Conflicts of Interest
None Declared.

Multimedia Appendix 1
STROBE Checklist.
[PDF File (Adobe PDF File), 150 KB-Multimedia Appendix 1]

Multimedia Appendix 2
Q-Q-Plots.
[PDF File (Adobe PDF File), 86 KB-Multimedia Appendix 2]

References
1.

2.
3.

4.

5.

6.

7.
8.
9.

10.

11.

12.

13.

Gopal G, Suter-Crazzolara C, Toldo L, Eberhardt W. Digital transformation in healthcare - architectures of present andfuture information technologies. Clin Chem Lab Med. 2019;57(3):328-335. [FREE Full text] [doi: 10.1515/cclm-2018-0658][Medline: 30530878]
Mangiapane M, Bender M. EMR Adoption Model (EMRAM). In: Mangiapane M, Bender M, editors. Patientenorientierte
Digitalisierung im Krankenhaus. Wiesbaden. Springer Vieweg; 2020;33-39.
Jacob PD. Chapter 3 - Management of patient healthcare information: healthcare-related information flow, access, andavailability. In: Gogia S, Novaes M, Basu A, Gogia K, Gogia S, editors. Fundamentals of Telemedicine and Telehealth.
London. Academic Press; 2020;35-57.
Embi PJ, Weir C, Efthimiadis EN, Thielke SM, Hedeen AN, Hammond KW. Computerized provider documentation:
findings and implications of a multisite study of clinicians and administrators. J Am Med Inform Assoc. 2013;20(4):718-726.
[FREE Full text] [doi: 10.1136/amiajnl-2012-000946] [Medline: 23355462]
Vos JFJ, Boonstra A, Kooistra A, Seelen M, van Offenbeek M. The influence of electronic health record use on collaboration
among medical specialties. BMC Health Serv Res. 2020;20(1):676. [FREE Full text] [doi: 10.1186/s12913-020-05542-6][Medline: 32698807]
Yanamadala S, Morrison D, Curtin C, McDonald K, Hernandez-Boussard T. Electronic health records and quality of care:an observational study modeling impact on mortality, readmissions, and complications. Medicine (Baltimore).
2016;95(19):e3332. [FREE Full text] [doi: 10.1097/MD.0000000000003332] [Medline: 27175631]
Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst. 2014;2:3. [FREEFull text] [doi: 10.1186/2047-2501-2-3] [Medline: 25825667]
Dick RS, Steen EB, Detmer DE. The Computer-Based Patient Record: An Essential Technology for Health Care, Revised
Edition. Washington. National Academies Press; 1997.
Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse forclinical research. J Am Med Inform Assoc. 2013;20(1):144-151. [FREE Full text] [doi: 10.1136/amiajnl-2011-000681][Medline: 22733976]
Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, et al. A harmonized data quality assessment
terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 2016;4(1):1244.
[FREE Full text] [doi: 10.13063/2327-9214.1244] [Medline: 27713905]
Wurster F, Fütterer G, Beckmann M, Dittmer K, Jaschke J, Köberlein-Neu J, et al. The analyzation of change in documentation
due to the introduction of electronic patient records in hospitals: a systematic review. J Med Syst. 2022;46(8):54. [FREEFull text] [doi: 10.1007/s10916-022-01840-0] [Medline: 35781136]
Montagna S, Croatti A, Ricci A, Agnoletti V, Albarello V, Gamberini E. Real-time tracking and documentation in trauma
management. Health Informatics J. 2020;26(1):328-341. [FREE Full text] [doi: 10.1177/1460458219825507] [Medline:
30726161]
Zargaran E, Spence R, Adolph L, Nicol A, Schuurman N, Navsaria P, et al. Association between real-time electronic injury
surveillance applications and clinical documentation and data acquisition in a South African trauma center. JAMA Surg.
2018;153(5):e180087. [FREE Full text] [doi: 10.1001/jamasurg.2018.0087] [Medline: 29541765]

https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
14.

15.

16.

17.

18.

19.
20.

21.

22.
23.
24.

25.
26.

27.
28.

29.

30.

31.
32.
33.
34.

35.

Thoroddsen A, Ehnfors M, Ehrenberg A. Content and completeness of care plans after implementation of standardized
nursing terminologies and computerized records. Comput Inform Nurs. 2011;29(10):599-607. [FREE Full text] [doi:10.1097/NCN.0b013e3182148c31] [Medline: 22041791]
McCamley J, Vivanti A, Edirippulige S. Dietetics in the digital age: The impact of an electronic medical record on a tertiary
hospital dietetic department. Nutr Diet. 2019;76(4):480-485. [FREE Full text] [doi: 10.1111/1747-0080.12552] [Medline:
31199071]
Karp EL, Freeman R, Simpson KN, Simpson AN. Changes in efficiency and quality of nursing electronic health record
documentation after implementation of an admission patient history essential data set. Comput Inform Nurs.
2019;37(5):260-265. [FREE Full text] [doi: 10.1097/CIN.0000000000000516] [Medline: 31094915]
Meier-Diedrich E, Davidge G, Hägglund M, Kharko A, Lyckblad C, McMillan B, et al. Changes in documentation due topatient access to electronic health records: protocol for a scoping review. JMIR Res Protoc. 2023;12:e46722. [FREE Fulltext] [doi: 10.2196/46722] [Medline: 37639298]
Wiebe N, Varela LO, Niven DJ, Ronksley PE, Iragorri N, Quan H. Evaluation of interventions to improve inpatient hospital
documentation within electronic health records: a systematic review. J Am Med Inform Assoc. 2019;26(11):1389-1400.
[FREE Full text] [doi: 10.1093/jamia/ocz081] [Medline: 31365092]
Emekli E, Coscun Ö, Budakoglu I, Kiyak YS. Clinical record keeping education needs in a medical school and the qualityof clinical documentations. Konuralp Med J. 2023;15(2):257-265. [FREE Full text] [doi: 10.18521/ktd.1259969]
von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. STROBE Initiative. The strengthening
the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies.
Ann Intern Med. 2007;147(8):573-577. [FREE Full text] [doi: 10.7326/0003-4819-147-8-200710160-00010] [Medline:
17938396]
Beckmann M, Dittmer K, Jaschke J, Karbach U, Köberlein-Neu J, Nocon M, et al. Electronic patient record and its effects
on social aspects of interprofessional collaboration and clinical workflows in hospitals (eCoCo): a mixed methods study
protocol. BMC Health Serv Res. 2021;21(1):377. [FREE Full text] [doi: 10.1186/s12913-021-06377-5] [Medline: 33892703]
Referenzdatenbank der Qualitätsberichte der Krankenhäuser. Gemeinsamer Bundesausschuss. 2023. URL: https://qb-referenzdatenbank.g-ba.de/ [accessed 2023-12-22]
Prior L. Repositioning documents in social research. Sociology. 2008;42(5):821-836. [FREE Full text] [doi:10.1177/0038038508094564]
Ranegger R, Hackl WO, Ammenwerth E. Implementation of the Austrian Nursing Minimum Data Set (NMDS-AT): a
feasibility study. BMC Med Inform Decis Mak. 2015;15:75. [FREE Full text] [doi: 10.1186/s12911-015-0198-7] [Medline:
26384111]
Toney-Butler TJ, Unison-Pace WJ. Nursing Admission Assessment and Examination. Treasure Island (FL). StatPearls
Publishing; 2018.
Yadav S, Kazanji N, Narayan KC, Paudel S, Falatko J, Shoichet S, et al. Comparison of accuracy of physical examination
findings in initial progress notes between paper charts and a newly implemented electronic health record. J Am Med Inform
Assoc. 2017;24(1):140-144. [FREE Full text] [doi: 10.1093/jamia/ocw067] [Medline: 27357831]
Muallem YA, Dogether MA, Househ M, Saddik B. Auditing the completeness and legibility of computerized radiologicalrequest forms. J Med Syst. 2017;41(12):199. [FREE Full text] [doi: 10.1007/s10916-017-0826-0] [Medline: 29101478]
Coffey C, Wurster LA, Groner J, Hoffman J, Hendren V, Nuss K, et al. A comparison of paper documentation to electronic
documentation for trauma resuscitations at a level i pediatric trauma center. J Emerg Nurs. 2015;41(1):52-56. [FREE Fulltext] [doi: 10.1016/j.jen.2014.04.010] [Medline: 24996509]
Seroussi B, Bouaud J. The (Re)-Relaunching of the DMP, the French shared medical record: new features to improve
uptake and use. In: Ugon A, Karlsson D, Klein GO, editors. Building Continents of Knowledge in Oceans of Data: The
Future of Co-Created EHealth. Amsterdam. IOS Press; 2018;256-260.
Jang J, Yu SH, Kim CB, Moon Y, Kim S. The effects of an electronic medical record on the completeness of documentationin the anesthesia record. Int J Med Inform. 2013;82(8):702-707. [FREE Full text] [doi: 10.1016/j.ijmedinf.2013.04.004][Medline: 23731825]
Brown ML. Can't you just pull the data? The limitations of using of the electronic medical record for research. Paediatr
Anaesth. 2016;26(11):1034-1035. [FREE Full text] [doi: 10.1111/pan.12951] [Medline: 27747978]
Pruitt Z, Pracht E. Upcoding emergency admissions for non-life-threatening injuries to children. Am J Manag Care.
2013;19(11):917-924. [FREE Full text] [Medline: 24511988]
Gesner E, Gazarian P, Dykes P. The burden and burnout in documenting patient care: an integrative literature review. Stud
Health Technol Inform. 2019;264:1194-1198. [doi: 10.3233/SHTI190415] [Medline: 31438114]
Saravi BM, Asgari Z, Siamian H, Farahabadi EB, Gorji AH, Motamed N, et al. Documentation of medical records in
Hospitals of Mazandaran University of medical sciences in 2014: a quantitative study. Acta Inform Med. 2016;24(3):202-206.
[FREE Full text] [doi: 10.5455/aim.2016.24.202-206] [Medline: 27482136]
Batko K, Ślęzak A. The use of big data analytics in healthcare. J Big Data. 2022;9(1):3. [FREE Full text] [doi:10.1186/s40537-021-00553-4] [Medline: 35013701]

https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

Wurster et al

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
36.

37.

38.

39.

40.
41.

Wurster et al

Ommaya AK, Cipriano PF, Hoyt DB, Horvath KA, Tang P, Paz HL, et al. Care-Centered clinical documentation in thedigital environment: solutions to alleviate burnout. National Academy of Medicine. 2018. URL: https://nam.edu/
care-centered-clinical-documentation-digital-environment-solutions-alleviate-burnout/ [accessed 2023-12-22]
Kapsner LA, Kampf MO, Seuchter SA, Gruendner J, Gulden C, Mate S, et al. Reduced rate of inpatient hospital admissions
in 18 German University Hospitals during the COVID-19 lockdown. Front Public Health. 2020;8:594117. [FREE Full text][doi: 10.3389/fpubh.2020.594117] [Medline: 33520914]
Videokonferenz der Bundeskanzlerin mit den Regierungschefinnen und Regierungschefs der Länder am 28. Oktober 2020.
Presse- und Informationsamt der Bundesregierung. 2020. URL: https://www.bundesregierung.de/resource/blob/997532/
1805024/5353edede6c0125ebe5b5166504dfd79/2020-10-28-mpk-beschluss-corona-data.pdf?download=1 [accessed2023-12-22]
Curtis CA, Nguyen MU, Rathnasekara GK, Manderson RJ, Chong MY, Malawaraarachchi JK, et al. Impact of electronic
medical records and COVID-19 on adult goals-of-care document completion and revision in hospitalised general medicine
patients. Intern Med J. 2022;52(5):755-762. [FREE Full text] [doi: 10.1111/imj.15543] [Medline: 34580964]
Wilcox RR. Introduction to Robust Estimation and Hypothesis Testing. Amsterdam. Academic Press; 2011.
Prokosch HU, Ganslandt T. Perspectives for medical informatics. reusing the electronic medical record for clinical research.
Methods Inf Med. 2009;48(1):38-44. [Medline: 19151882]

Abbreviations
EMR: electronic medical record
HIS: hospital information system
ICD: International Classification of Diseases
STROBE: Strengthening the Reporting of Observational Studies in Epidemiology

Edited by J Hefner; submitted 31.03.23; peer-reviewed by S Veeranki, N Nekliudov, C Hudak; comments to author 23.07.23; revised
version received 10.08.23; accepted 23.10.23; published 19.01.24
Please cite as:
Wurster F, Beckmann M, Cecon-Stabel N, Dittmer K, Hansen TJ, Jaschke J, Köberlein-Neu J, Okumu MR, Rusniok C, Pfaff H, KarbachU
The Implementation of an Electronic Medical Record in a German Hospital and the Change in Completeness of Documentation:
Longitudinal Document Analysis
JMIR Med Inform 2024;12:e47761
URL: https://medinform.jmir.org/2024/1/e47761
doi: 10.2196/47761
PMID: 38241076©Florian Wurster, Marina Beckmann, Natalia Cecon-Stabel, Kerstin Dittmer, Till Jes Hansen, Julia Jaschke, Juliane Köberlein-Neu,
Mi-Ran Okumu, Carsten Rusniok, Holger Pfaff, Ute Karbach. Originally published in JMIR Medical Informatics(https://medinform.jmir.org), 19.01.2024. This is an open-access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete
bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license
information must be included.

https://medinform.jmir.org/2024/1/e47761

XSL• FO
RenderX

JMIR Med Inform 2024 | vol. 12 | e47761 | p. 11(page number not for citation purposes)