JMIR MEDICAL INFORMATICS

Heo et al

Original Paper

Time Series AI Model for Acute Kidney Injury Detection
Based on a Multicenter Distributed Research Network:
Development and Verification Study
Suncheol Heo1*, MS; Eun-Ae Kang2*, BA; Jae Yong Yu1*, PhD; Hae Reong Kim1, PhD; Suehyun Lee3, PhD;
Kwangsoo Kim4, PhD; Yul Hwangbo5, MD, PhD; Rae Woong Park6, MD, PhD; Hyunah Shin7, BA; Kyeongmin
Ryu7, BA; Chungsoo Kim8, PharmD; Hyojung Jung5, BA; Yebin Chegal9, BA; Jae-Hyun Lee10,11, MD, PhD; Yu
Rang Park1, PhD
1

Department of Biomedical System Informatics, Yonsei University College of Medicine, Seoul, Republic of Korea
Medical Informatics Collaborative Unit, Department of Research Affairs, Yonsei University College of Medicine, Seoul, Republic of Korea
3
Department of Computer Engineering, Gachon University, Seongnam, Republic of Korea
4
Transdisciplinary Department of Medicine & Advanced Technology, Seoul National University Hospital, Seoul, Republic of Korea
5
Healthcare AI Team, National Cancer Center, Goyang, Republic of Korea
6
Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Republic of Korea
7
Healthcare Data Science Center, Konyang University Hospital, Daejeon, Republic of Korea
8
Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Republic of Korea
9
Department of Statistics, Korea University, Seoul, Republic of Korea
10
Division of Allergy and Immunology, Department of Internal Medicine, Yonsei University College of Medicine, Seoul, Republic of Korea
11
Institute of Allergy, Yonsei University College of Medicine, Seoul, Republic of Korea*these authors contributed equally2

Corresponding Author:
Yu Rang Park, PhD
Department of Biomedical System Informatics
Yonsei University College of Medicine
50-1 Yonsei-ro, Seodaemun-gu
Seoul, 03722
Republic of Korea
Phone: 82 2-2228-2493
Email: yurangpark@yuhs.ac

Abstract
Background: Acute kidney injury (AKI) is a marker of clinical deterioration and renal toxicity. While there are many studies
offering prediction models for the early detection of AKI, those predicting AKI occurrence using distributed research network(DRN)–based time series data are rare.
Objective: In this study, we aimed to detect the early occurrence of AKI by applying an interpretable long short-term memory(LSTM)–based model to hospital electronic health record (EHR)–based time series data in patients who took nephrotoxic drugs
using a DRN.
Methods: We conducted a multi-institutional retrospective cohort study of data from 6 hospitals using a DRN. For each
institution, a patient-based data set was constructed using 5 drugs for AKI, and an interpretable multivariable LSTM (IMVLSTM) model was used for training. This study used propensity score matching to mitigate differences in demographics andclinical characteristics. Additionally, the temporal attention values of the AKI prediction model’s contribution variables were
demonstrated for each institution and drug, with differences in highly important feature distributions between the case andcontrol data confirmed using 1-way ANOVA.
Results: This study analyzed 8643 and 31,012 patients with and without AKI, respectively, across 6 hospitals. When
analyzing the distribution of AKI onset, vancomycin showed an earlier onset (median 12, IQR 5-25 days), and acyclovir
was the slowest compared to the other drugs (median 23, IQR 10-41 days). Our temporal deep learning model for AKI
prediction performed well for most drugs. Acyclovir had the highest average area under the receiver operating characteristic
curve score per drug (0.94), followed by acetaminophen (0.93), vancomycin (0.92), naproxen (0.90), and celecoxib (0.89).
https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et al

Based on the temporal attention values of the variables in the AKI prediction model, verified lymphocytes and calcvancomycin
ium had the highest attention, whereas lymphocytes, albumin, and hemoglobin tended to decrease over time, and urine pH andprothrombin time tended to increase.
Conclusions: Early surveillance of AKI outbreaks can be achieved by applying an IMV-LSTM based on time series data
through an EHR-based DRN. This approach can help identify risk factors and enable early detection of adverse drug reactions
when prescribing drugs that cause renal toxicity before AKI occurs.
JMIR Med Inform 2024;12:e47693; doi: 10.2196/47693
Keywords: adverse drug reaction; real world data; multicenter study; distributed research network; common data model; timeseries AI; time series; artificial intelligence; machine learning; adverse reaction; adverse reactions; detect; detection; toxic;
toxicity; renal; kidney; nephrology; pharmaceutical; pharmacology; pharmacy; pharmaceutics

Introduction

Methods

Acute kidney injury (AKI) is associated with a mortality rateof 40%‐70% in hospitalized patients who develop AKI andcauses significant kidney damage even after recovery, leading
to dialysis, longer hospital stays, and increased costs of care
[1-4]. Early detection of AKI increases the likelihood of AKI
prevention, associated morbidity, and costs [5]. As no specifictreatment can reverse AKI and the recognition of patientsat risk of AKI before diagnosis contributes to better clinicaloutcomes than treatment after AKI occurs [6], early detectionof AKI is essential for prompt therapeutic intervention.

Ethical Considerations

Several studies have attempted to predict AKI occurrence.
With the increasing availability of clinical databases, they
developed models to predict the occurrence of AKI using
electronic health records (EHRs) [7-18]. Although thesestudies used EHRs, the number of patients in the patient
population was small because they focused on specificpatients, such as surgical patients, patients with sepsis, andolder adults. There have also been a number of studies
using artificial intelligence (AI) models to predict AKI.
Although attempts have been made to predict the occurrenceof AKI early, few models have provided clear rationales andexplanations [19-21]. Therefore, time series data analysis isrequired for AKI prediction models to reflect the temporal
information between variables [22]. Time series analysis for
AKI is necessary because the length of time each patient stays
in a hospital or intensive care unit can differ from person toperson, and the frequency of measurements can vary fromvalues that are measured continuously (eg, blood pressure)
to laboratory values that are measured on an as-neededbasis. Recently, an interpretable multivariable long short-term
memory (IMV-LSTM) method for considering time series
data has been published [23]; however, little research has
been conducted on this method.
To address these issues, this study aimed to apply andvalidate a multicenter-based explainable time series AI model
for predicting the occurrence of AKI caused by specificnephrotoxic drugs in 6 hospitals in South Korea by using alarge clinical database with a common data model (CDM)
through a distributed research network (DRN).

https://medinform.jmir.org/2024/1/e47693

This study was approved by the institutional review committees of Severance Hospital (SH; 4-2021-1209), Gangnam Severance Hospital (GSH; 3-2021-0005), Konyang
University Hospital (KYUH; 2021-10-003-001), Ajou
University Hospital (AJUH; AJIRB-MED-MDB-21‐676),
Seoul National University Cancer Hospital (SNUH;
E-2207-151-1342), and the National Cancer Center (NCC;
2022-0184). All retrospective data were anonymized andappropriate measures were taken to protect participant
information.

Study Design
This retrospective observational cohort study analyzed the
EHRs from 6 hospitals in South Korea between 1994
and 2021 to predict AKI. The EHRs were converted to
OMOP-CDM (Observational Medical Outcomes Partnership
Common Data Model) version 5.3.1. The 6 hospitals wereSH, GSH, KYUH, AJUH, SNUH, and the NCC. An overall
diagram of the cohort composition is provided in Figure 1.
In our cohorts, we adopted criteria based on serum
creatinine (SCr) to define AKI according to the Kidney
Disease: Improving Global Outcomes (KDIGO) Clinical
Practice Guidelines and the previously defined AKI classification stages mapped in the “injury” category [24-26]. The
criterion is an increase in the SCr level to 2 times the baseline
value. As an alternative to the baseline SCr levels, we defined
the upper limit of normal (ULN) value of SCr as 1.2 mg/dl[27].
The targeted drugs were selected from 5 medications
associated with a high risk of AKI according to the US
Food and Drug Administration (FDA) and previous studies:acetaminophen; vancomycin; 2 nonsteroidal anti-inflammatory drugs (NSAIDs), naproxen and celecoxib; and 1 antiviraldrug, acyclovir [21,22].
The inclusion criteria were as follows: (1) the target drugs
were administered, (2) the patient had a visit record of at least
30 days prior to the observation period, and (3) the patientunderwent at least 2 SCr tests during the preobservation
period (0‐60 days before the study). The exclusion criteria
were as follows: (1) the patient had at least 1 SCr test outside
the ULN value in the preobservation period. Participants were
JMIR Med Inform 2024 | vol. 12 | e47693 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
divided into case and control cohorts based on whether they
met previously defined AKI criteria for 60 days after taking
the first medicine. The observation period refers to the time
range before and after the initial medication intake for eachpatient within the cohort. The cohort definitions were createdusing ATLAS, a web-based tool (Observational Health Data
Sciences and Informatics), and are available as JSON files on
GitHub [28].

Heo et alpropensity score matching (PSM). Covariates included wereage, sex, and SCr value at baseline. We normalized thecovariates by applying standard scaling to ensure consistentdimensions across variables. A propensity score for eachpatient was generated by logistic regression. Patients werematched in a 1:3 ratio using a K-nearest neighbor (K-NN)
algorithm using the Python scikit-learn library.

To adjust for differences between cases and controls
to reduce the effect of confounding variables, we used
Figure 1. Overall flowchart for predicting acute kidney injury events. AJUH: Ajou University Hospital; GSH: Gangnam Severance Hospital;
IMV-LSTM: interpretable multivariable long short-term memory; KYUH: Konyang University Hospital; NCC: National Cancer Center; OMOP:
Observational Medical Outcomes Partnership; SH: Severance Hospital; SNUH: Seoul National University Cancer Hospital; ULN, upper limit of
normal.

Candidate Predictors for Time Series
Candidate predictors were extracted from several key
domains within the CDM containing per-patient observational
data using structured query language tools in Python. Ageand sex were used in the person domain, clinical laboratory tests in the measurement domain, medications in the
drug exposure domain, diagnostic records in the condition
occurrence domain, and surgical/procedure records in theprocedure occurrence domain. Lab tests were treated ascontinuous variables, while other medications, conditions,
https://medinform.jmir.org/2024/1/e47693

and procedures were treated as binary variables. Statistical methods were used to select the variables. To identify predictors, we tested the statistical significance of thedifference between the enrollment time of the cohort andthe onset date of AKI using a 2-tailed paired t test and the
McNemar test for continuous and dichotomous variables,
respectively. To create a time series table, the candidate
variables were pivoted into columns and dates were placed
into rows. Missing values were handled in the following
ways: forward fill for laboratory tests and diagnoses and
JMIR Med Inform 2024 | vol. 12 | e47693 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et alzero fill for medications and treatments. The window size forpredictions used 4-week sequence data and was processed byshifting the data of the prediction cycle by 2 weeks.

Results

AKI Prediction Modeling

The demographics of the 31,012 patients without AKI and the8643 patients with AKI across the 6 hospitals after PSM are
shown in Table 1.

LSTM models based on recurrence have been designed
to process time series data [29]. Attention-based LSTM
models were initially proposed for learning words and therelationships between words in natural language processing[30,31] and later evolved into a key component of deeplearning, becoming one of the methods used to provide
interpretations, including importance scores for predicted
outcomes.
As an advanced LSTM model, we used the IMV-LSTM
module for the learning model, which is a multivariate
LSTM neural network for the prediction and interpretationof multivariate time series [23]. This model improves on the
LSTM-attention model, which can predict variable importance using multivariate inputs to configure variable-wisehidden states and mix both temporal and variable levelsof attention for improved interpretability. The model wastrained for 200 epochs with a batch size of 64 and a learning rate of 1e-3. An Adam optimizer was used with early
stopping after 20 epochs. The data set was divided intotraining, test, and validation sets at a 6:2:2 ratio. Prediction
performance was evaluated using the area under the receiver
operating characteristic curve (AUROC) value. Additionally,
we used the accuracy, precision, F1-score, and area under
the precision-recall curve (AUPRC) to ensure robustness forunbalanced data.
In this study, AKI prediction models were created for eachhospital and drug. Each model had a different selection ofcandidate variables. To interpret the predictors in each model,
variable- and temporal-wise attention scores were extracted
from all trained models. These scores were then aggregatedby calculating the overall temporal attention score, which
was obtained by taking the weighted average of the temporalattention value over the attention value for each predictor
variable. The resulting scores were plotted as heat maps forinterpretation.

Statistical Analysis
We used statistical packages based on Python and R (RProject for Statistical Computing) for the statistical analysis.
First, to compare the AKI and non-AKI groups, we calculated significance using the χ2 test for categorical variables
and an independent-sample 2-tailed t test for continuous
variables. Second, to identify differences in the pattern of
AKI occurrence between cohorts and drugs, a histogram wasplotted for patients in each cohort from the date of cohort
entry (the first day of administration of the target drug) to
the date of AKI occurrence. Differences between drugs wereanalyzed using an independent-sample2-tailed t test. Third,
we compared the distribution of the aggregated temporalattention scores with the actual trained data with box plots ofthe data for 4 weeks at 1-week intervals. A repeated ANOVA
test was performed to identify temporal differences.
https://medinform.jmir.org/2024/1/e47693

Demographic and Clinical Characteristics

By hospital, the cohort consisted of 14,046 patients fromSH, 2180 from GSH, 2493 from KYUH, 4331 from AJUH,
13,174 from SNUH, and 4431 from NCC after matching thepropensity scores. Propensity matching was performed using
age, sex, and SCr levels at baseline. As for the changes incovariates, the difference in mean age decreased from 6.39
(60.65 – 54.26) to 0.44 (60.65 – 60.21) years, the differencein male ratio decreased from 16.49% (63.17% – 46.68%)
to 7.02% (63.17% – 56.15%), and the difference in SCr atbaseline decreased from 0.14 mg/dL (0.71 – 0.58 mg/dL)
to 0.03 mg/dL (0.71 – 0.67 mg/dL) (Table S2 in Multimedia Appendix 1). There were still statistically significant
differences in PSM age (60.66, SD 15.86 vs 60.22, SD 15.94
years; P=.03), gender (63.17% vs 56.15% male; P<.001), andSCr at baseline (0.71, SD 0.61 vs 0.68, SD 0.41 mg/dL;
P<.001). Patients who developed AKI had more severe
neoplasms (ie, active cancers; 70.6% vs 44.26%; P<.001) andchronic liver disease (15.83% vs 6.55%; P<.001). Moreover, the analysis revealed the following differences: sepsis(6.5% vs 1.94%; P<.001), diabetes mellitus (19.43% vs14.57%; P<.001), hypertension (27.81% vs 23.5%; P<.001),
anemia (9.81% vs 4.8%; P<.001), and heart failure (4.79%
vs 3.07%; P<.001). There was no significant difference
between hypotension (0.24% vs 0.21%; P=.56), potassium(4.11, SD 0.57 vs 4.11, SD 0.51 mmol/L; P=.48), or renalartery stenosis (0.07% vs 0.03%; P=.13). Hypoalbuminemia,
obesity, peripheral vascular disease, renal artery stenosis,
liver dysfunction, and prior kidney surgery had low incidence
rates (<2%).

Distribution of Adverse Drug Events
To analyze the differences in drug patterns at the time of
AKI occurrence, we assessed the pattern for each drug. The
median number of days for the occurrence of AKI among
the drug and cohort patients in the entire hospital was 17(IQR 7-33 days). Vancomycin appeared after a median periodof 12 days, followed by naproxen (18 days), acetaminophen(19 days), celecoxib (22 days), and acyclovir (23 days).
When comparing the IQR values, celecoxib (10‐41 days) and
Acyclovir (10‐41 days) showed a relatively broad distribution, whereas acetaminophen (9‐34 days) and naproxen (8‐34days) were distributed over 25 days. Vancomycin (5‐25days) exhibited the narrowest distribution. Celecoxib andacyclovir tended to be relatively distributed compared toacetaminophen, vancomycin, and naproxen. We compared theonset times of all drug pairs and hospital pairs to check thesimilarity in AKI occurrence (Figure 2). The patterns between
specific drugs was similar for celecoxib and acyclovir (P=.88)
and for acetaminophen and naproxen (P=.57). The patterns
between hospitals were similar for SH and AJUH (P=.98), SH
and GSH (P=.36), GSH and AJUH (P=.42), GSH and NCC(P=.24), and SNUH and NCC (P=.26).
JMIR Med Inform 2024 | vol. 12 | e47693 | p. 4
(page number not for citation purposes)

https://medinform.jmir.org/2024/1/e47693

56(11.41)Chronic liver disease, 462
n (%)(15.26)0541(17.87)62 (2.05)

2 (0.2)652 (65)

2 (0.37)

40 (3.99)

19 (3.53) 18 (1.79)

0

255(47.31)0

1 (0.19)

3 (0.3)25 (5.09) 35 (6.49) 89 (8.87)

2 (0.41)

9 (1.83)

2 (0.41)320(65.17)

Laboratory values (before medication), mean (SD)Prior kidney surgery,n (%)

Anemia, n (%)

Liver dysfunction, n(%)

3 (0.1)

2112(69.75)25 (0.83)

Obesity, n (%)

Neoplasm (active
cancers), n (%)

Peripheral vasculardisease, n (%)

351 (35)

1 (0.1)0266 (8.78) 13 (2.65) 34 (6.31) 33 (3.29)

77(14.29)00Heart failure, n (%)

179(36.46)4 (0.81)2 (0.41)52 (9.65) 139(13.86)24 (4.45) 13 (1.3)

223(22.23)139(13.86)375(37.39)628(62.61)(n=1003)
59.52(15.79)

AJUHd

1462
(48.28)12 (0.4)

7 (0.23)142 (4.69) 3 (0.61)

65(12.06)7 (1.3)179(33.21)360(66.79)(n=539)67.91(13.64)KYUHc

Hypertension, n (%)

Hypotension, n (%)

Hypoalbuminemia, n(%)

Chronic kidneydisease, n (%)

50(10.18)117(23.83)325(10.73)915(30.22)

Sepsis, n (%)

Diabetes mellitus, n(%)

205(41.75)286(58.25)1063(35.11)

1965(64.89)(n=491)
62.6(15.04)

GSHb

FemaleMale

Age (years), mean
(SD)
Gender, n (%)(n=3028)
61.83(15.23)

SHa

Case group

2 (0.21)00

6 (0.62)581(60.14)14 (0.54)05512(50.03)4178(37.92)30 (0.27)

9 (0.08)51 (0.46)

36 (0.33)

18 (0.21)

0848 (9.81) 903 (8.2)

139 (1.61) 71 (0.64)

81 (0.94)

10 (0.12)414 (4.79) 629 (5.71)

6001(69.43)21 (0.24)

9 (0.1)71 (7.35) 2404
(27.81)00766 (6.95)

2462(22.35)373 (3.39)

3926(35.63)7092(64.37)SH(n=11,018)
61.14(15.37)

Control group202 (2.34) 271 (2.46)

16 (1.66) 1231
(14.24)2 (0.21)562 (6.5)

3183(36.83)5460(63.17)

Total(n=8643)60.65(15.5)

59 (6.11) 1679
(19.43)1 (0.1)397(41.1)

569(58.9)(n=966)60.55(12.24)

NCCf

154 (5.89) 4 (0.41)

31 (1.19)

10 (0.38)

3 (0.11)62 (2.37)

2081(79.55)264(10.09)4 (0.15)0506(19.34)18 (0.69)

300(11.47)40 (1.53)964(36.85)1652(63.15)(n=2616)57.89(16.49SNUHe

053 (3.14)

4 (0.24)

26 (1.54)

3 (0.18)

20 (1.18)

531(31.44)455(26.94)3 (0.18)

2 (0.12)82 (4.85)

8 (0.47)266(15.75)37 (2.19)

704(41.68)985(58.32)GSH(n=1689)
62.26(14.81)0

68 (3.48)

25 (1.28)

69 (3.53)

4 (0.2)40 (0.42)

3 (0.09)1282(38.52)

1143(34.34)4 (0.12)0

59 (0.62)

37 (0.39)

50 (0.52)

3 (0.09)

2 (0.06)00

11 (0.12)

1 (0.03)176 (5.29) 282 (2.95) 7 (0.2)

46 (1.38)

35 (1.05)

10 (0.3)

2470(71.28)127 (1.33) 10 (0.29)

4450(46.56)00

12 (0.04)

1489 (4.8)

208 (0.67)

220 (0.71)

103 (0.33)

951 (3.07)

14,662(47.28)65 (0.21)

11 (0.04)

924 (9.67) 229 (6.61) 7289 (23.5)

28 (0.29)

01846 (5.95)

384 (1.24)

601 (1.94)

13,600(43.85)

17,412(56.15)Total(n=31,012)
60.21(15.61)836 (8.75) 167 (4.82) 4519
(14.57)1 (0.03)1447(41.76)2018(58.24)NCC(n=3465)
60.14(12.71)126 (3.79) 759 (7.94) 42 (1.21)

11 (0.33)

105 (5.37) 60 (1.8)

417(21.34)360(18.42)00

71 (3.63)

51 (2.61)

4956(51.85)4602(48.15)SNUH(n=9558)
57.46(16.61)128 (3.85) 41 (0.43)

1563(46.97)1765(53.03)

AJOU(n=3328)59.67(15.77)195 (9.98) 593(17.82)21 (1.07)

1004(51.38)950(48.62)KYUH(n=1954)67.71(13.55)

<.001

<.001

<.001

.03

<.001

<.001

<.001

<.001

.56

.01

<.001

<.001

<.001

<.001

<.001

<.001

.02

P value

Table 1. Demonstration and clinical characteristics of patients with and without AKI across 6 hospitals after propensity score matching. The P values were obtained by conducting a 2-sample t test to compare
the means for cases and controls.

JMIR MEDICAL INFORMATICS
Heo et al

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 5
(page number not for citation purposes)

138.6(4.16)
17.83(9.66)

Sodium(mmol/L)
BUN(bloodurea
nitrogen)(mg/dL)https://medinform.jmir.org/2024/1/e47693

137.27(4.36)
15.44(6.58)

4.21(0.47)

132.1(52.11)0.79(0.23)(n=491)

GSHb

136.49(4.61)
17.67(8.87)

4.14(0.6)

150.75(82.36)0.99(0.37)(n=539)KYUHc

aSH: Severance Hospital.
bGSH: Gangnam Severance Hospital.
cKYUH: Konyang University Hospital.
dAJUH: Ajou University Hospital.
eSNUH: Seoul National University Cancer Hospital.
fNCC: National Cancer Center.

Potassium 3.92 (0.6)(mmol/L)

126.89(58.33)

Glucose(mg/dL)Serum
0.46
creatinine (0.74)(mg/dL)(n=3028)

SHa

Case group138.48(4.23)
14.68(6.36)

4.17(0.55)

187.0(23.0)

0.87(0.37)(n=1003)

AJUHd

138.46(3.96)
14.72(7.61)

4.22 (0.5)

121.48(48.26)0.87(0.47)(n=2616)

SNUHe

138.31(3.67)
15.55(6.13)

4.29(0.56)

125.85(50.05)1.01(0.29)(n=966)

NCCf

138.3
(4.14)
16.13(8.28)

4.11(0.57)

126.93(56.28)0.75(0.59)

Total(n=8643)

4.22 (0.4)

124.3(49.43)0.76(0.19)

GSH(n=1689)139.44138.31(3.75)(3.58)
16.82 (7.48) 15.78(5.86)

3.93 (0.55)

121.72(55.7)

0.44 (0.51)SH(n=11,018)

Control group137.74(3.73)
17.19(7.62)

4.09(0.54)

140.11(78.81)0.98(0.22)

KYUH(n=1954)139.27(3.85)
14.9(6.41)

4.15(0.46)

151.5(39.5)

0.84 (0.2)

AJOU(n=3328)139.71(3.23)
14.67(5.14)

4.22(0.43)

115.65(40.5)

0.83(0.18)

SNUH(n=9558)

138.96(3.37)
14.63(4.96)

4.28(0.52)

120.88(46.23)0.98(0.16)

NCC(n=3465)

.48

<.001

<.001

15.67 (6.47) <.001

139.28 (3.6) <.001

4.11 (0.51)

120.95(51.85)0.72 (0.4)

Total(n=31,012)

P value

JMIR MEDICAL INFORMATICS
Heo et al

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et al

Figure 2. (A) Comparison of acute kidney injury (AKI) onset time between drugs and (B) AKI onset time between hospitals. The P values wereobtained by conducting independent 2-tailed t tests between each aggregated pair. AJUH: Ajou University Hospital; GSH: Gangnam Severance
Hospital; KYUH: Konyang University Hospital; NCC: National Cancer Center; SH: Severance Hospital; SNUH: Seoul National University Cancer
Hospital.

AKI Prediction Model Performance
The AUROC for each drug and hospital to evaluate the
AKI predictive model, based on respective test sets (internal
validation) is shown in Figure 3. A total of 26 trained models
achieved a high AUROC value, of 0.92 on average, with eachverification data set. In addition, among the averages of thedrugs, acyclovir had the highest average AUROC score of
0.94, followed by acetaminophen (0.93), vancomycin (0.92),
naproxen (0.90), and celecoxib (0.89). The highest AUROC
value (0.97) was observed for the model of SH’s celecoxib
and acyclovir, SNUH’s vancomycin, and KYUH’s acyclovirprescription patients.

https://medinform.jmir.org/2024/1/e47693

Multimedia Appendix 2 presents data on the precision,accuracy, F1-score, and AUPRC of each predictive model.
Overall, the average accuracy of the AKI prediction models
was 0.88, whereas the average AUPRC and F1-scores wereboth 0.78. The acyclovir prescription model achieved thehighest accuracy score (0.91), followed by vancomycin(0.90), acetaminophen (0.89), naproxen (0.89), and celecoxib(0.86). Individually, the acyclovir SH model showed the best
performance, with an AUPRC of 0.92 and an accuracy of
0.91.

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et al

Figure 3. Receiver operating characteristic (ROC) curves and areas under the curve (AUCs) of the acute kidney injury prediction model for eachhospital and each drug. The square brackets indicate the 95% CI. AJUH: Ajou University Hospital; GSH: Gangnam Severance Hospital; KYUH:
Konyang University Hospital; NCC: National Cancer Center; SH: Severance Hospital; SNUH: Seoul National University Cancer Hospital.

Temporal Feature Importance of the AKI
Prediction Model
To interpret the AKI prediction model, we demonstrated thetemporal attention values of each contributing variable in the
4 weeks prior to AKI onset, which were weighted aggregatesfrom the model for each drug and hospital, as shown in Figure
4A. The temporal change pattern of the actual data corresponding to each variable in the 4 weeks prior to AKI onset
is shown in Figure 4B. We also confirmed the difference inthe distribution of highly important features between the caseand control data using a 1-way ANOVA. The attention scoresfor all variables across all hospitals are detailed in Multimedia
Appendix 3

https://medinform.jmir.org/2024/1/e47693

The last week of lymphocytes (attention score at −1 week:0.41) and the second week of calcium (attention score at −3
weeks: 0.41) showed the highest attention scores, followed byalbumin (attention score at −1 week: 0.37; attention score at−4 weeks: 0.37), hemoglobin (attention score at −4 weeks:0.37), and cholesterol (attention score at −4 weeks: 0.37).
In Figure 5, the distribution of data by variable between
the 2 groups was confirmed using actual data. There was adifference in data distribution between the case and control
groups from the beginning. The values of lymphocytes,
albumin, and hemoglobin in the case group decreased overtime, while urine pH and prothrombin time in the case grouptended to increase over time.

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et al

Figure 4. (A) Temporal attention score of important features of acute kidney injury (AKI) prediction model and (B) distribution of data over time (Pvalues: repeated measures ANOVA). The figure shows the change over a 4-week period prior to the AKI event.

https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et al

Figure 5. Acute kidney injury (AKI) onset time after drug administration at various medical centers. The red lines shows the median value. AJUH’s
naproxen and GSH’s acyclovir cases were excluded as the number of AKI case groups was less than 20. AJUH: Ajou University Hospital; GSH:
Gangnam Severance Hospital; KYUH: Konyang University Hospital; NCC: National Cancer Center; SH: Severance Hospital; SNUH: Seoul National
University Cancer Hospital.

Discussion
Principal Findings
In this study, we developed time series–based IMV-LSTM
models to predict AKI in patients taking specific nephrotoxic
drugs using CDM-based DRNs in a 6-hospital EHR-based
system. The principal findings are as follows: first, this study
provides an interpretation of the temporal importance ofvariables for predicting AKI, and the models also achieved
high performance, with an average AUC of 0.92%. Second,
our study is a scalable multicenter study using a DRN,
which can contribute to understanding drug-induced AKI. To
the best of our knowledge, this is the first study to build
an AKI prediction model by applying a time series–based
IMV-LSTM model to a CDM using EHR data from 6
hospitals.
We established a retrospective cohort of patients who took
nephrotoxicity-inducing drugs at 6 hospitals. With respect
to demographic characteristics, we observed variations in
the overall patient count and prevalence of comorbidities
when comparing individuals with AKI and without AKI
across different hospitals. Nevertheless, the majority ofpatients who developed AKI at most hospitals were older
than 60 years and had a high prevalence of comorbidities,
including cancer (n=6001, 69.43%), hypertension (n=2404,
27.81%), diabetes (n=1679, 19.43%), and chronic liverdisease (n=1231, 14.24%), which is consistent with findings
reported in previous studies [32-35].
https://medinform.jmir.org/2024/1/e47693

The pattern of each drug’s association with AKI (Figure5) showed that the median number of days for AKI onset
when using nephrotoxic drugs was 17 (IQR 7-33) days. The
onset occurred earliest with vancomycin (12, IQR 5-25 days)
and latest with acyclovir (23, IQR 10-41 days). In previous
studies [36,37], the time to onset of vancomycin-induced
AKI showed a similar pattern to our results. We also founddifferences in the AKI onset between different classes within
the same NSAID, and the multicenter AKI cohort showedsimilarities between hospitals. The finding of similar patterns
in the AKI onset in the multicenter cohort supports thereliability of the AKI cohort and increases the explanatory
power of AKI prediction models.
AKI is common among inpatients [38,39]. Previous
models predicted AKI in the intensive care unit (ICU) andin surgical patients during hospital admission. For example,
Zimmerman et al [40] predicted the occurrence of AKI in
ICU inpatients (AUC 0.783), and Tseng et al [41] developed
a predictive score for the development of AKI after cardiac
surgery (AUC 0.839). Hsu et al [42] developed a risk scorefunction for community-acquired AKI for inpatients (AUC
0.818). Koyner et al [10] developed a model to predict AKI in
hospitalized patients (AUC 0.90). Despite this progress, few
studies have applied time series deep learning to provide a
temporal interpretation of drug-induced AKI, and our model
stands out because it can predict nephrotoxic drug–induced
AKI in a diverse hospital population.

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICSThis study achieved improved performance compared toprevious AKI studies using recurrent neural networks (RNNs)[21,43-45]. Our model improves performance up to an AUC
of 0.97 and an overall average of 0.92, which outperforms
previous studies showing AKI prediction with RNN-based
methods by Kim et al [43] in hospitalized patients (AUC
0.927), Rank et al [45] in cardiac surgery patients (AUC
0.893), and Xu et al [44] in inpatients (AUC 0.908). These
results show promise for our model as a tool to predict AKI
and facilitate early intervention and mitigation strategies forpatients.
This study also provides additional interpretations
regarding the temporal importance of features for AKI
prediction. Some studies have reported information oninterpretability or provided information about the interpretability of variables at the feature importance level [46].
However, our results show the importance of variables andthe temporal importance of variables in the development of
AKI. In this study, we highlight the vital role of temporal
patterns of various indicators, such as lymphocytes, calcium,
albumin, hemoglobin, and cholesterol, in predicting diseasestates, particularly the onset of AKI. The temporal patternof lymphocytes increased gradually, peaking 1 week before
AKI onset. The use of lymphocyte and neutrophil counts aspredictive factors for AKI is consistent with other studies[47,48]. Calcium shows a pattern of peaking 3 weeks before
AKI onset, and Prior studies showed an association between
impacted calcium metabolism and AKI [49,50]. Albuminshows the highest pattern 1 and 4 weeks before onset, andlow serum albumin levels (hypoalbuminemia) are a predictorof AKI [48,51,52]. Hemoglobin shows a pattern with a peak
4 weeks before onset, and previous studies have shown thatthe risk of AKI increases stepwise with a further decreasein hemoglobin concentration [53]. Temporal variations invariables based on reported laboratory data for the early
detection of AKI emphasize the importance of monitoring andearly intervention in populations.

Heo et al
In addition, this retrospective study can be followed bya subsequent study to validate the practicality of the AKI
prediction model in clinical practice by applying it to a
prediction system in a hospital EHR.

Limitations
This study has several limitations. First, because we used
the CDM, it does not reflect the full range of clinical data.
For example, we could not include admission records, which
would have revealed a patient’s condition. However, the useof CDM data allowed for a multicenter study that could be
easily extended to other institutions that have converted to theCDM. Second, this was a retrospective study and could not
address the underlying causes of AKI. Therefore, prospective studies are needed for validation with actual clinicaldata. Third, as with all retrospective studies, there may
be unintentional patient selection bias and unaccounted-for
confounders. However, to compensate for these limitations,
we tried to equalize the distribution of patient characteristics
through PSM. We also used a limited follow-up period tominimize the impact of these factors.

Conclusions
This study demonstrates the high performance of the IMVLSTM method for AKI prediction using hospital EHR-based
time series data. Our model can provide real-time assessmentof AKI occurrence and individualized risk factors for AKI
using time series data. We also demonstrated the robustnessof our model through multicenter validation using a CDM
through a DRN of 6 hospitals in South Korea, which also
proves that scalability to other institutions that are converted to the CDM is possible. This may provide an objective
quantitative tool for identifying patients at risk of developing
AKI.

Acknowledgments
This study was supported by a grant from the Korea Institute of Drug Safety and Risk Management in 2021. This work was
supported by the Bio-Industrial Technology Development Program (20014841) funded by the Ministry of Trade, Industry &Energy (Korea). We would like to thank the Medical Informatics Collaborative Unit members of Yonsei University College ofMedicine for their assistance in data analysis.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Demographics for each institution.
[XLSX File (Microsoft Excel File), 60 KB-Multimedia Appendix 1]
Multimedia Appendix 2
Model performance.
[XLSX File (Microsoft Excel File), 11 KB-Multimedia Appendix 2]
Multimedia Appendix 3
Feature importance.
[XLSX File (Microsoft Excel File), 68 KB-Multimedia Appendix 3]
https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 11(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et alReferences
1.
Chertow GM, Burdick E, Honour M, Bonventre JV, Bates DW. Acute kidney injury, mortality, length of stay, and costs
in hospitalized patients. J Am Soc Nephrol. Nov 2005;16(11):3365-3370. [doi: 10.1681/ASN.2004090740] [Medline:
16177006]
2.
Waikar SS, Curhan GC, Ayanian JZ, Chertow GM. Race and mortality after acute renal failure. J Am Soc Nephrol. Oct
2007;18(10):2740-2748. [doi: 10.1681/ASN.2006091060] [Medline: 17855647]
3.
Kee YK, Kim EJ, Park KS, et al. The effect of specialized continuous renal replacement therapy team in acute kidney
injury patients treatment. Yonsei Med J. May 2015;56(3):658-665. [doi: 10.3349/ymj.2015.56.3.658] [Medline:
25837170]
4.
Medina KRP, Jeong JC, Ryu JW, et al. Comparison of outcomes of mild and severe community- and hospital-acquired
acute kidney injury. Yonsei Med J. Oct 2022;63(10):902-907. [doi: 10.3349/ymj.2021.0238] [Medline: 36168242]
5.
Kate RJ, Perez RM, Mazumdar D, Pasupathy KS, Nilakantan V. Prediction and detection models for acute kidney injury
in hospitalized older adults. BMC Med Inform Decis Mak. Mar 29, 2016;16:39. [doi: 10.1186/s12911-016-0277-4][Medline: 27025458]
6.
Khwaja A. KDIGO Clinical Practice Guidelines for acute kidney injury. Nephron Clin Pract. 2012;120(4):c179-c184.
[doi: 10.1159/000339789] [Medline: 22890468]
7.
Ugwuowo U, Yamamoto Y, Arora T, et al. Real-time prediction of acute kidney injury in hospitalized adults:
implementation and proof of concept. Am J Kidney Dis. Dec 2020;76(6):806-814. [doi: 10.1053/j.ajkd.2020.05.003][Medline: 32505812]
8.
Low S, Vathsala A, Murali TM, et al. Electronic health records accurately predict renal replacement therapy in acutekidney injury. BMC Nephrol. Jan 31, 2019;20(1):32. [doi: 10.1186/s12882-019-1206-4] [Medline: 30704418]
9.
Chua HR, Zheng K, Vathsala A, et al. Health care analytics with time-invariant and time-variant feature importance topredict hospital-acquired acute kidney injury: observational longitudinal study. J Med Internet Res. Dec 24,
2021;23(12):e30805. [doi: 10.2196/30805] [Medline: 34951595]
10. Koyner JL, Carey KA, Edelson DP, Churpek MM. The development of a machine learning inpatient acute kidney injury
prediction model. Crit Care Med. Jul 2018;46(7):1070-1077. [doi: 10.1097/CCM.0000000000003123] [Medline:
29596073]
11. Wilson FP, Greenberg JH. Acute kidney injury in real time: prediction, alerts, and clinical decision support. Nephron.
2018;140(2):116-119. [doi: 10.1159/000492064] [Medline: 30071528]
12. Huang CY, Güiza F, De Vlieger G, et al. Development and validation of clinical prediction models for acute kidney
injury recovery at hospital discharge in critically ill adults. J Clin Monit Comput. Feb 2023;37(1):113-125. [doi: 10.
1007/s10877-022-00865-7] [Medline: 35532860]
13. Bedford M, Stevens P, Coulton S, et al. Development of risk models for the prediction of new or worsening acute kidney
injury on or during hospital admission: a cohort and nested study. Health Serv Deliv Res. 2016;4(6):1-160. [doi: 10.
3310/hsdr04060] [Medline: 26937542]
14. Kim WH, Lee SM, Choi JW, et al. Simplified clinical risk score to predict acute kidney injury after aortic surgery. J
Cardiothorac Vasc Anesth. Dec 2013;27(6):1158-1166. [doi: 10.1053/j.jvca.2013.04.007] [Medline: 24050856]
15. Kiers HD, van den Boogaard M, Schoenmakers MCJ, et al. Comparison and clinical suitability of eight prediction
models for cardiac surgery-related acute kidney injury. Nephrol Dial Transplant. Feb 2013;28(2):345-351. [doi: 10.1093/
ndt/gfs518] [Medline: 23222415]
16. Zhou LZ, Yang XB, Guan Y, et al. Development and validation of a risk score for prediction of acute kidney injury inpatients with acute decompensated heart failure: a prospective cohort study in China. J Am Heart Assoc. Nov 16,
2016;5(11):e004035. [doi: 10.1161/JAHA.116.004035] [Medline: 27852590]
17. Wilson T, Quan S, Cheema K, et al. Risk prediction models for acute kidney injury following major noncardiac surgery:systematic review. Nephrol Dial Transplant. Feb 2016;31(2):231-240. [doi: 10.1093/ndt/gfv415] [Medline: 26705194]
18. Sanchez-Pinto LN, Khemani RG. Development of a prediction model of early acute kidney injury in critically ill children
using electronic health record data. Pediatr Crit Care Med. Jun 2016;17(6):508-515. [doi: 10.1097/PCC.
0000000000000750] [Medline: 27124567]
19. Sutherland SM, Chawla LS, Kane-Gill SL, et al. Utilizing electronic health records to predict acute kidney injury risk
and outcomes: workgroup statements from the 15th ADQI Consensus Conference. Can J Kidney Health Dis. 2016;3:11.
[doi: 10.1186/s40697-016-0099-4] [Medline: 26925247]
20. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models
instead. Nat Mach Intell. May 2019;1(5):206-215. [doi: 10.1038/s42256-019-0048-x] [Medline: 35603010]
21. Dong J, Feng T, Thapa-Chhetry B, et al. Machine learning model for early prediction of acute kidney injury (AKI) in
pediatric critical care. Crit Care. Aug 10, 2021;25(1):288. [doi: 10.1186/s13054-021-03724-0] [Medline: 34376222]
https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 12(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
22.
23.
24.
25.

26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.

Heo et al

Lee JM, Hauskrecht M. Modeling multivariate clinical event time-series with recurrent temporal mechanisms. Artif Intell
Med. Feb 2021;112:102021. [doi: 10.1016/j.artmed.2021.102021] [Medline: 33581828]
Guo T, Lin T, Antulov-Fantulin N, editors. Exploring interpretable LSTM neural networks over multi-variable data. In:
Proceedings of the 36th International Conference on Machine Learning. PMLR; 2019.
Kellum JA, Lameire N, Aspelin P, et al. Kidney disease: improving global outcomes (KDIGO) acute kidney injury workgroup. KDIGO Clinical Practice Guideline for acute kidney injury. Kidney Int Suppl. 2012;2(1):1-138. [doi: 10.1038/
kisup.2012.1]
Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, Acute Dialysis Quality Initiative workgroup. Acute renal
failure - definition, outcome measures, animal models, fluid therapy and information technology needs: the Second
International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care. Aug
2004;8(4):R204-R212. [doi: 10.1186/cc2872] [Medline: 15312219]
Mehta RL, Kellum JA, Shah SV, et al. Acute Kidney Injury Network: report of an initiative to improve outcomes inacute kidney injury. Crit Care. 2007;11(2):R31. [doi: 10.1186/cc5713] [Medline: 17331245]
Hosten AO. BUN and creatinine. In: Clinical Methods: The History, Physical, and Laboratory Examinations. 3rd ed.
Butterworths; 1990.
Development and verification of a time series AI model for acute kidney injury detection based on a multicenter
distributed research network. GitHub. URL: https://github.com/DigitalHealthcareLab/22MOACDM [Accessed2023-07-08]
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. Nov 15, 1997;9(8):1735-1780. [doi: 10.1162/
neco.1997.9.8.1735] [Medline: 9377276]
Luong MT, Pham H, Manning CD. Effective approaches to attention-based neural machine translation. arXiv. Preprint
posted online on Sep 20, 2015. [doi: 10.48550/arXiv.1508.04025]
Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv. Preprint
posted online on Sep 1, 2014. [doi: 10.48550/arXiv.1409.0473]
Harirforoosh S, Asghar W, Jamali F. Adverse effects of nonsteroidal antiinflammatory drugs: an update ofgastrointestinal, cardiovascular and renal complications. J Pharm Pharm Sci. 2013;16(5):821-847. [doi: 10.18433/
j3vw2f] [Medline: 24393558]
Perazella MA. Drug-induced nephropathy: an update. Expert Opin Drug Saf. Jul 2005;4(4):689-706. [doi: 10.1517/
14740338.4.4.689] [Medline: 16011448]
Khan S, Loi V, Rosner MH. Drug-induced kidney injury in the elderly. Drugs Aging. Oct 2017;34(10):729-741. [doi: 10.
1007/s40266-017-0484-4] [Medline: 28815461]
Finlay S, Bray B, Lewington AJ, et al. Identification of risk factors associated with acute kidney injury in patients
admitted to acute medical units. Clin Med (Lond). Jun 2013;13(3):233-238. [doi: 10.7861/clinmedicine.13-3-233][Medline: 23760694]
van Hal SJ, Paterson DL, Lodise TP. Systematic review and meta-analysis of vancomycin-induced nephrotoxicity
associated with dosing schedules that maintain troughs between 15 and 20 milligrams per liter. Antimicrob Agents
Chemother. Feb 2013;57(2):734-744. [doi: 10.1128/AAC.01568-12] [Medline: 23165462]
Filippone EJ, Kraft WK, Farber JL. The nephrotoxicity of vancomycin. Clin Pharmacol Ther. Sep 2017;102(3):459-469.
[doi: 10.1002/cpt.726] [Medline: 28474732]
Kate RJ, Pearce N, Mazumdar D, Nilakantan V. A continual prediction model for inpatient acute kidney injury. Comput
Biol Med. Jan 2020;116:103580. [doi: 10.1016/j.compbiomed.2019.103580] [Medline: 32001013]
Kellum JA, Romagnani P, Ashuntantang G, Ronco C, Zarbock A, Anders HJ. Acute kidney injury. Nat Rev Dis Primers.
Jul 15, 2021;7(1):52. [doi: 10.1038/s41572-021-00284-z] [Medline: 34267223]
Zimmerman LP, Reyfman PA, Smith ADR, et al. Early prediction of acute kidney injury following ICU admission using
a multivariate panel of physiological measurements. BMC Med Inform Decis Mak. Jan 31, 2019;19(Suppl 1):16. [doi:10.1186/s12911-019-0733-z] [Medline: 30700291]
Tseng PY, Chen YT, Wang CH, et al. Prediction of the development of acute kidney injury following cardiac surgery bymachine learning. Crit Care. Jul 31, 2020;24(1):478. [doi: 10.1186/s13054-020-03179-9] [Medline: 32736589]
Hsu CN, Liu CL, Tain YL, Kuo CY, Lin YC. Machine learning model for risk prediction of community-acquired acutekidney injury hospitalization from electronic health records: development and validation study. J Med Internet Res. Aug
4, 2020;22(8):e16903. [doi: 10.2196/16903] [Medline: 32749223]
Kim K, Yang H, Yi J, et al. Real-time clinical decision support based on recurrent neural networks for in-hospital acutekidney injury: external validation and model interpretation. J Med Internet Res. Apr 16, 2021;23(4):e24120. [doi: 10.
2196/24120] [Medline: 33861200]

https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 13(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.

Heo et al

Xu J, Hu Y, Liu H, et al. A novel multivariable time series prediction model for acute kidney injury in general
hospitalization. Int J Med Inform. May 2022;161:104729. [doi: 10.1016/j.ijmedinf.2022.104729] [Medline: 35279551]
Rank N, Pfahringer B, Kempfert J, et al. Deep-learning-based real-time prediction of acute kidney injury outperforms
human predictive performance. NPJ Digit Med. 2020;3:139. [doi: 10.1038/s41746-020-00346-8] [Medline: 33134556]
Wang Y, Wei Y, Yang H, Li J, Zhou Y, Wu Q. Utilizing imbalanced electronic health records to predict acute kidney
injury by ensemble learning and time series model. BMC Med Inform Decis Mak. Sep 21, 2020;20(1):238. [doi: 10.
1186/s12911-020-01245-4] [Medline: 32957977]
Weller S, Varrier M, Ostermann M. Lymphocyte function in human acute kidney injury. Nephron. 2017;137(4):287-293.
[doi: 10.1159/000478538] [Medline: 28662513]
Ishikawa M, Iwasaki M, Namizato D, et al. The neutrophil to lymphocyte ratio and serum albumin as predictors of acutekidney injury after coronary artery bypass grafting. Sci Rep. Sep 14, 2022;12(1):15438. [doi: 10.1038/s41598-02219772-7] [Medline: 36104386]
Wang B, Li D, Gong Y, Ying B, Cheng B. Association of serum total and ionized calcium with all-cause mortality in
critically ill patients with acute kidney injury. Clinica Chimica Acta. Jul 2019;494:94-99. [doi: 10.1016/j.cca.2019.03.
1616]
Thongprayoon C, Cheungpasitporn W, Mao MA, Sakhuja A, Erickson SB. Admission calcium levels and risk of acutekidney injury in hospitalised patients. Int J Clin Pract. Apr 2018;72(4):e13057. [doi: 10.1111/ijcp.13057] [Medline:
29314467]
Wiedermann CJ, Wiedermann W, Joannidis M. Hypoalbuminemia and acute kidney injury: a meta-analysis ofobservational clinical studies. Intensive Care Med. Oct 2010;36(10):1657-1665. [doi: 10.1007/s00134-010-1928-z][Medline: 20517593]
Thongprayoon C, Cheungpasitporn W, Mao MA, Sakhuja A, Kashani K. U-shape association of serum albumin level
and acute kidney injury risk in hospitalized patients. PLoS One. 2018;13(6):e0199153. [doi: 10.1371/journal.pone.
0199153] [Medline: 29927987]
Walsh M, Garg AX, Devereaux PJ, Argalious M, Honar H, Sessler DI. The association between perioperative
hemoglobin and acute kidney injury in patients having noncardiac surgery. Anesth Analg. Oct 2013;117(4):924-931.
[doi: 10.1213/ANE.0b013e3182a1ec84] [Medline: 24023017]

Abbreviations
ADR: adverse drug reaction
AI: artificial intelligence
AJUH: Ajou University Hospital
AKI: acute kidney injury
AUPRC: area under the precision-recall curve
AUROC: area under the receiver operating characteristic curve
CDM: common data model
DRN: distributed research network
EHR: electronic health record
FDA: Food and Drug Administration
GSH: Gangnam Severance Hospital
ICU: intensive care unit
IMV-LSTM: interpretable multivariable long short-term memory
KDIGO: Kidney Disease Improving Global Outcomes
KYUH: Konyang University Hospital
MIMIC: Medical Information Mart for Intensive Care
NCC: National Cancer Center
NSAID: nonsteroidal anti-inflammatory drug
OMOP-CDM: Observational Medical Outcomes Partnership Common Data Model
PSM: propensity score matching
RNN: recurrent neural network
SCr: serum creatinine
SH: Severance Hospital
SNUH: Seoul National University Cancer Hospital
ULN: upper limit of normal

https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 14(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Heo et al

Edited by Christian Lovis; peer-reviewed by Marek Oja, Xiangzhou Zhang; submitted 30.03.2023; final revised version
received 08.07.2023; accepted 19.05.2024; published 05.07.2024
Please cite as:
Heo S, Kang EA, Yu JY, Kim HR, Lee S, Kim K, Hwangbo Y, Park RW, Shin H, Ryu K, Kim C, Jung H, Chegal Y, Lee JH,
Park YR
Time Series AI Model for Acute Kidney Injury Detection Based on a Multicenter Distributed Research Network: Development and Verification StudyJMIR Med Inform 2024;12:e47693
URL: https://medinform.jmir.org/2024/1/e47693
doi: 10.2196/47693© Suncheol Heo, Eun-Ae Kang, Jae Yong Yu, Hae Reong Kim, Suehyun Lee, Kwangsoo Kim, Yul Hwangbo, Rae Woong
Park, Hyunah Shin, Kyeongmin Ryu, Chungsoo Kim, Hyojung Jung, Yebin Chegal, Jae-Hyun Lee, Yu Rang Park. Originallypublished in JMIR Medical Informatics (https://medinform.jmir.org), 05.07.2024. This is an open-access article distributed
under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits
unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical
Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.

https://medinform.jmir.org/2024/1/e47693

JMIR Med Inform 2024 | vol. 12 | e47693 | p. 15(page number not for citation purposes)