JMIR AI

Tao et al

Original Paper

Predictive Modeling of Hypertension-Related Postpartum
Readmission: Retrospective Cohort Analysis
Jinxin Tao1, PhD; Ramsey G Larson2, MD; Yonatan Mintz1, PhD; Oguzhan Alagoz1, PhD; Kara K Hoppe3, MS, DO
1

Industrial and Systems Engineering, University of Wisconsin Madison, Madison, WI, United States

2

Department of Obstetrics and Gynecology, MultiCare Rockwood Clinic, Spokane, WA, United States

3

Department of Obstetrics and Gynecology, School of Medicine and Public Health, University of Wisconsin Madison, Madison, WI, United States

Corresponding Author:
Yonatan Mintz, PhD
Industrial and Systems Engineering
University of Wisconsin Madison
3107 Mechanical Engineering Building
1513 University Avenue
Madison, WI, 53706
United States
Phone: 1 6092180891
Email: ymintz@wisc.edu

Abstract
Background: Hypertension is the most common reason for postpartum hospital readmission. Better prediction of postpartum
readmission will improve the health care of patients. These models will allow better use of resources and decrease health carecosts.
Objective: This study aimed to evaluate clinical predictors of postpartum readmission for hypertension using a novel machinelearning (ML) model that can effectively predict readmissions and balance treatment costs. We examined whether blood pressureand other measures during labor, not just postpartum measures, would be important predictors of readmission.
Methods: We conducted a retrospective cohort study from the PeriData website data set from a single midwestern academic
center of all women who delivered from 2009 to 2018. This study consists of 2 data sets; 1 spanning the years 2009-2015 andthe other spanning the years 2016-2018. A total of 47 clinical and demographic variables were collected including blood pressure
measurements during labor and post partum, laboratory values, and medication administration. Hospital readmissions were verified
by patient chart review. In total, 32,645 were considered in the study. For our analysis, we trained several cost-sensitive ML
models to predict the primary outcome of hypertension-related postpartum readmission within 42 days post partum. Models were
evaluated using cross-validation and on independent data sets (models trained on data from 2009 to 2015 were validated on thedata from 2016 to 2018). To assess clinical viability, a cost analysis of the models was performed to see how their recommendations
could affect treatment costs.
Results: Of the 32,645 patients included in the study, 170 were readmitted due to a hypertension-related diagnosis. A cost-sensitive
random forest method was found to be the most effective with a balanced accuracy of 76.61% for predicting readmission. Usinga feature importance and area under the curve analysis, the most important variables for predicting readmission were bloodpressures in labor and 24-48 hours post partum increasing the area under the curve of the model from 0.69 (SD 0.06) to 0.81 (SD0.06), (P=.05). Cost analysis showed that the resulting model could have reduced associated readmission costs by US $6000
against comparable models with similar F1-score and balanced accuracy. The most effective model was then implemented as arisk calculator that is publicly available. The code for this calculator and the model is also publicly available at a GitHub repository.
Conclusions: Blood pressure measurements during labor through 48 hours post partum can be combined with other variables
to predict women at risk for postpartum readmission. Using ML techniques in conjunction with these data have the potential to
improve health outcomes and reduce associated costs. The use of the calculator can greatly assist clinicians in providing care topatients and improve medical decision-making.
(JMIR AI 2024;3:e48588) doi: 10.2196/48588

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

JMIR AI 2024 | vol. 3 | e48588 | p. 1
(page number not for citation purposes)

JMIR AITao et al

KEYWORDS
pregnancy; postpartum; hypertension; preeclampsia; blood pressure; hospital readmission, clinical calculator; healthcare cost;
cost; cohort analysis; utilization; resources; labor; women; risk; readmission; cohort; hospital; statistical model; retrospective
cohort study; predict; risk

Introduction
Hypertensive disorders of pregnancy (HDP) are common andestimated to occur in 10% of pregnancies in the United States.
In addition to complicating the management of pre- andpostdelivery periods, hypertension is the leading cause of
postpartum readmission, accounting for 9.3%-27% of
postpartum readmissions [1-3]. Postpartum readmission iscostly, both in health care dollars and in quality-of-life measures
for mothers and new families. In addition, HDP increases
maternal morbidity and mortality and is associated with anincreased risk of cardiovascular disease later in life [4-8].
A study published in the Journal of Hypertension in 2018
estimated that preventable postpartum readmission in womenwith hypertension resulted in 20,240 excess inpatient hospitaldays and US $36 million excess medical costs [9]. Rates andreasons for readmission have been under recent scrutiny andoffer an area to improve health care delivery and preventativecare. All-cause hospital readmission rates are on the rise withrisk factors for all-cause postpartum readmission including
public insurance, race, presence of comorbid conditions
including hypertension and diabetes, and cesarean section [3].
Approximately 30% of women who experience
hypertension-related postpartum readmissions do not haveantecedent diagnoses of hypertension, thus making it imperative
to include normotensive patients without an HDP before
postpartum discharge in evaluating for postpartum readmission[10]. As such, our objective was to identify key clinicalvariables, in addition to demographic characteristics, implicatedin postpartum readmission of all birthing persons using amachine learning (ML) model. This prediction task ischallenging because while costs related to readmission are high,readmission rates are low resulting in highly imbalanced data
sets that are challenging to use in training ML models. For
instance, while existing models have strong overall accuracy
performance (out of sample, an area under the curve of 0.81)[11], they do so by trading off high specificity for low sensitivity
which could result in many readmission cases going undetected
and not properly treated. We hypothesized that blood pressure
metrics during labor, not just post partum, would impact
readmission rates. Similarly, we hypothesized thatantihypertensive medication administration and highpreeclampsia laboratory values during initial readmission would
increase the readmission rate.

Methods
Ethical Considerations
We obtained institutional review board approval (#2016-006).
Individual patient consent was not required due to theretrospective study design. The data set was deidentified before
study analysis. No compensation was provided to human
https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

participants as this was a retrospective study that involved
development of a retrospective data set using electronic medical
records (EMRs).

Chart Review and Inclusion
We initially performed a retrospective chart review of all
patients who delivered at a single, midwestern academic center
hospital between 2009 and 2015. Inclusion criteria for this study
included all women who delivered a baby in this time frame.
We wanted to ensure that we captured all hypertension-related
readmissions within 42 days post partum regardless of adiagnosis of hypertension before hospital discharge from thedelivery of their infant. The primary outcome was
hypertension-related readmission; therefore, all readmissions
included in this data set were specific to hypertension only. To
confirm our previous results and create a larger sample size, we
extended the study population to include births from 2016 to2018 and used a similar process to manage the data of all
patients who delivered at the same birthing hospital. We used
the hospital’s PeriData website [12] data set, which is used tocontribute birth-related outcomes to the state-wide database forclinical perinatal information and additional hospital-run reports
to obtain additional data available from the EMR. We collected
demographic as well as clinical data, including blood pressure
measurements during labor and post partum, laboratory data,
and medication administration at the patient level to be ourpredictor variables. Hospital readmissions and our prediction
response variable were verified by patient chart review. Given
that the data came from multiple sources and had missing
observations, the raw data set could not be used directly for
analysis.

Analytics Plan
Data Processing and Feature Engineering
We processed the raw data set and then merged the processeddata including patient demographics, blood pressure
measurements, medication administration, and laboratory
information from different sources into 1 pandas data frame[13]. Race and ethnicity were entered in the medical chart basedon the patient’s self-identity at the time of admission to thehealth care system. Laboratory results were included in thisanalysis because they are involved in the classification andseverity of HDP. Laboratory results included liver function
tests, hemoglobin and platelet counts, creatinine, and urine
protein. We analyzed blood pressure records with timestamps
and identified the highest systolic blood pressure and associateddiastolic blood pressure during 3 time periods, that are, labor,
0-24 hours post partum, and 24-48 hours post partum, because
we expected blood pressure during labor and post partum to beimportant features for predicting hypertensive readmissions.
Using the medication administration data from the EMR, weconstructed the following binary (yes or no) attributes for thefollowing medication name and route administered: (1) oral
labetalol, (2) intravenous labetalol, (3) oral nifedipine-immediate
JMIR AI 2024 | vol. 3 | e48588 | p. 2
(page number not for citation purposes)

JMIR AI
release, (4) oral nifedipine-extended release, (5) intravenous
hydralazine, and (6) oral ibuprofen. To obtain these features,
we started with a full medication data set for each patient’s
medical registration number that included the medication name,time administered, dosage, and route of administration. This
meant that there were multiple entries per medical registration
number if a particular patient was given that medication morethan once. The key challenge of using the medication data wasthat there were significant missing data; moreover, not all
patients received all medications, and the individual medication
schedules could be infrequent. For this reason, we considered
only the binary attributes instead of the full medication schedule
to ensure that the data points were dense enough for the analysis.

Predictive Model Training and Validation
We used a cost-sensitive random forest method to predict which
patient would experience a hypertension-related postpartum
readmission [14]. Since the data set was imbalanced (only 170
readmissions out of 32,645 participants), the use of class weightsthat penalize false negatives significantly higher than false
positives was necessary to avoid ML models that predict every
sample as the negative class. We considered other candidateclassifiers namely logistic regression with L1 or L2
regularization, support vector machines (SVM) with polynomial,radial basis function or sigmoid kernel, and a standard decision
tree approach for the prediction task. To measure the predictive
performance of each model, we considered a combination ofdifferent metrics. In the case of imbalanced data, reporting highaccuracy may be inappropriate since a highly accurate model
could simply ignore the rare class and still achieve highaccuracy. Therefore, we considered 2 complementary scoresfor assessing our model namely balanced accuracy and theF1-score [15]. The balanced accuracy can be thought of asbalancing the frequency of true positives and true negatives.
When calculating accuracy, it can be calculated by averaging
the true negative rate (specificity) and the true positive rate(sensitivity) of the model. In addition to prediction accuracy,
since our setting has a low frequency of positive cases, we
needed to ensure our selected model had high precision(alternatively low false alarm rate). For that reason, we also
considered the F1-score, which measures the balance between
the precision and the true positive rate. We tuned thehyperparameters of each model using cross-validation. For theouter loop, we iterated every hyperparameter combination. Thenwe performed stratified 5-fold cross-validation in the inner loop
and optimized the hyperparameters by evaluating the averagebalanced accuracy. Each model was trained using its respectiveclassifier implementation from scikit-learn [16].
We trained models on the 2009-2015 data and 2016-2018 data
individually and validated them using the 5-fold cross-validation

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

Tao et alpipeline. The purpose of this was to see if different factors
impacted readmission rates and decisions between the 2 time
periods. For added validation, we computed the performanceof models trained on the 2009-2015 data set using the 2016-2018
data to evaluate our pipeline. The final model deployed inpractice was tuned using the combined data set and 5-foldcross-validation. We performed a feature importance analysis
on the best models chosen by cross-validation for each data set.

Cost Analysis and Estimating Clinical Impact
To estimate the clinical impact of predictions, we completed 2
different forms of cost analysis. For each candidate model
considered, we used the above cross-validation procedure tocompute their estimated implementation costs.
We estimated the value of a false negative (an unplanned or
unpredicted readmission) to be US $20,439 and the value of a
false positive (the price of labetalol for 6 weeks for a patient
who ultimately did not need it) to be US $36. These costs werebased on estimates derived from our previous research [17]. In
addition, for the cost-sensitive random forest model (which weultimately determined was the most effective model), we
performed an additional analysis. For this analysis, we took themodel’s score for how likely a patient was to be classified as
needing readmission and compared it with a predictive threshold.
If the model score was larger than the threshold, the model
would predict that the patient would be readmitted. When thethreshold is <0.5, more patients are predicted to be readmitted
and if the threshold is >0.5, more patients are predicted to not
be readmitted. We used leave-one-out cross-validation tocompute the overall medical costs and balanced accuracy for
different thresholds between 0 and 1. The goal of this analysis
was to see how model scores should be interpreted in practiceby decision makers so that overall medical costs are minimized.

Results
Data Overview
From January 2009 to December 2018, a total of 39,133 womendelivered at our hospital; however, only 32,645 had complete
medical records available for analysis. Of these, 170 women
were readmitted for a hypertension-related diagnosis. There
was a statistically significant difference between the readmitted
group and the not readmitted group in terms of maternal age,gestational age at delivery, race, BMI, mode of delivery, andhypertension diagnosis. The readmitted group was more likely
to be older, having earlier gestational age at delivery, Black
race, higher BMI, cesarean delivery, and having a diagnosis ofchronic or pregnancy-induced hypertension (Table 1). The rateof hypertension diagnosis in our sample was 9%. The rate ofreadmission was 0.5%.

JMIR AI 2024 | vol. 3 | e48588 | p. 3
(page number not for citation purposes)

JMIR AITao et al

Table 1. Patient demographics and comparisons between the readmitted group and the not readmitted group.
Characteristics

All patients (N=32,645)

Readmitted (n=170)

Not readmitted (n=32,475)

P value

Maternal age, mean (SD)30.5 (5.3)

32.9 (5.7)

30.5 (5.3)<.001

Parity, n (%).29

Nulliparous

10,786 (33)

61 (35.9)

10,725 (33)Multiparous

17,946 (55)

95 (55.9)

17,851 (55)

Unknown

3913 (12)14 (8.2)

3899 (12)37.7 (2.5)

38.9 (2.4)

Gestational age at delivery in weeks, mean 38.9 (2.4)(SD)
Race, n (%)

<.001

White

26,188 (80.2)

130 (76.5)

26,058 (80.2)

Black1221 (3.7)

18 (10.6)1203 (3.7)

Asian Indian2532 (7.8)

12 (7.1)

2520 (7.8)

Asian, other855 (2.6)

3 (1.8)852 (2.6)

American Indian or Native

457 (1.4)

1 (0.6)456 (1.4)

Native Hawaiian

67 (0.2)0 (0)67 (0.2)

Unknown or other1325 (4.1)

6 (3.5)1319 (4.1)

Hispanic, n (%)

.05

Yes

2937 (9)8 (4.7)2929 (9)No

29,708 (91)162 (95.3)

29,546 (91)26.5 (8.6)

28.7 (8.6)

26.5 (8.9)

BMIa, mean (SD)Mode of delivery, n (%)

.001
<.001

Vaginal20,217 (61.9)

76 (44.7)

20,141 (62)

Vaginal vacuum

1695 (5.2)

8 (4.7)1687 (5.2)

Vaginal forceps

582 (1.8)

3 (1.8)579 (1.8)

Cesarean section

10,151 (31.1)

83 (48.8)

10,068 (31)2952 (9)96 (56.5)2856 (8.8)

Without preeclampsia

299 (1)16 (9.4)283 (0.9)

With preeclampsia

284 (0.9)

15 (8.8)269 (0.8)

Gestational hypertension

793 (2.4)

13 (7.6)780 (2.4)

Mild

582 (1.8)

26 (15.3)

556 (1.7)

Severe828 (2.5)

18 (10.6)810 (2.5)

Unspecified

166 (0.5)

8 (4.7)158 (0.5)

Hypertension diagnosis, n (%)

<.001

<.001

Chronic hypertension

PreeclampsiaaBMI: weight in kilograms divided by the square of height in meters.

Predictive Model Results
During our initial analysis of the data from 2009 to 2015, weevaluated 47 clinical and demographic variables to assess theirimportance in predicting postpartum readmission (Figure 1).
The variables most important for predicting readmission
included blood pressure parameters during labor and throughthe postpartum period as well as factors such as prepregnancy
BMI, maternal age, and gestational age at delivery. Variables
that had less predictive value included an HDP, administration
https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderXof antihypertensive medication, and mode of delivery. To
increase the predictive accuracy of the model, many of thesevariables were excluded from the next analysis. Even with fewer
variables, again the diagnosis of HDP and mode of delivery
were of least importance and blood pressure data during laborand post partum were most important. Additional details on themodel feature importance and feature correlations can be foundin the Multimedia Appendix 1 [18]. Through cross-validation
analysis, we found that the best model in terms of balanced
accuracy and F1-score was the random forest model. We
JMIR AI 2024 | vol. 3 | e48588 | p. 4
(page number not for citation purposes)

JMIR AI
performed an additional validation by using the best-tuned
models from the 2009 to 2015 data set on the 2016 to 2018 data
sets; the results are shown in Table 2. Each model was trained
only on the 2009-2015 data and was used to predict readmission
for patients in the 2016-2018 data. As shown in Table 2, theTao et al
random forest model achieves the best-balanced accuracy andF1-scores among all candidate models. Please refer to theMultimedia Appendix 1 [18] for the full set of cross-validation
parameters for each model.

Figure 1. Feature importance plot for hypertension-related postpartum readmission, 2009-2015. APGAR: appearance, pulse, grimace response, activity,
respiration; DBP: diastolic blood pressure; NICU: neonatal intensive care unit; SBP: systolic blood pressure.

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

JMIR AI 2024 | vol. 3 | e48588 | p. 5
(page number not for citation purposes)

JMIR AITao et al

Table 2. Predictive performance of models trained using the 2009-2015 data set on the 2016 to 2018 data sets.

aModel typeSpecificity, %Sensitivity, %Precision or PPVa, %NPVb, %

F1-score

Balanced accuracy, % Cost (US $)Random forest

70.86

75.81

1.38

99.82

0.027

73.33

426,240

Decision tree

64.94

75.81

1.15

99.8

0.023

70.37

450,828

SVMc

53.21

90.32

1.03

99.9

0.020

71.77

316,512

Logistic regression L1 62.12

83.87

1.17

99.86

0.023

72.99

360,864

Logistic regression L2 63.09

83.87

1.21

99.86

0.024

73.48

356,832

PPV: positive predictive value.

b

NPV: negative predictive value.

c

SVM: support vector machines.

Feature Importance Analysis
The additional 11,608 participants from deliveries between 2016
and 2018 were then added to the data set, and medication
administration data and laboratory data were included in thenext analysis. The final data set included 32,645 patients. Outof 33,482 total patients, 837 were excluded from the analysis
because of incomplete information regarding key features. We
ranked the features by their predictive importance and selected
the final set of features to be (1) BMI; (2) gestational age atdelivery; (3) maternal age; (4) highest systolic blood pressureduring 3 time periods, that were labor, 0-24 hours post partum,and 24-48 hours post partum; and (5) binary medication features.
The laboratory features were discarded because of their low
predictive feature importance. The most important clinicalvariable in predicting readmission was systolic blood pressure

between 24 and 48 hours post partum, and the second mostimportant was systolic blood pressure during labor (Figure 2).
Other factors that continued to be of importance in predicting
readmission included gestational age at delivery, maternal age,and prepregnancy BMI. We computed the correlation between
our proposed features (Figure 3). The receiver operating
characteristic (ROC) curves demonstrate that our model is able
to distinguish between a true positive (meaning predicting areadmission) and a false positive (meaning incorrectly predicting
a readmission). To show the significance of blood pressure
features in readmission prediction, we did a ROC curve
comparison using a 10-fold cross-validation with and without
blood pressure features and calculated the mean ROC andassociated SD, respectively. We can see a significant decline
in the classification performance without blood pressure features(Figure 4).

Figure 2. Validated feature importance plot for hypertension-related postpartum readmission, 2009-2018 (blood pressure values were the highest
recorded values during the specified time frames).

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

JMIR AI 2024 | vol. 3 | e48588 | p. 6
(page number not for citation purposes)

JMIR AITao et al

Figure 3. Correlation matrix between features on the combined data from 2009 to 2018. IV: intravenous; PO: orally.

Figure 4. Receiver operating curve comparisons with and without blood pressure features including highest systolic blood pressure and associateddiastolic blood pressure during 3 time periods—labor, between 0-24 hours postpartum, and between 24-48 hours postpartum. AUC: area under the
curve.

Cost Analysis and Final Model Tuning
To tune and validate the final model deployed in a calculator,
we also evaluated the model by measuring the estimated healthcare costs associated with the predictions. As previously
mentioned, the value of a false negative was estimated to be
US $20,439, and the value of a false positive to be US $36. The
cost ratio was then created by dividing those 2 numbers andcame out at 565 [17]. Lowering or raising this cost ratio places
more weight on different factors; for example, the side effects
associated with taking labetalol versus the time away fromfamily or a job during a readmission. With this information, theestimated total cost for each model can be calculated by using
the numbers of false negatives and false positives in thevalidation sets to give a sense for the medical impact of model
implementation. However, there are a lot more factors that needhttps://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

to be considered if we want the metric to be as generalizable as
the balanced accuracy and the F1-score. For this reason, we did
not consider estimated cost as the primary metric for model
selection. In Table 3, we can see that of the models considered,
the random forest model with class weight 1:200 had the highestbalanced accuracy. Compared with the best logistic regression
models and SVM, random forest with class weight 1:200
performs slightly better in terms of both balanced accuracy andF1-score. However, random forest model with class weight of
1:500 can provide better precision and F1-score. Combining the
2 metrics, we decided to implement the random forest modelwith class weight 1:500. Note that for all models the F1-scoreis relatively low, this is mainly due to the large imbalance inthe data set. Since readmissions are fairly rare, to ensure that
we avoid false negatives, we must reduce the precision of themodel leading to the reduced score.
JMIR AI 2024 | vol. 3 | e48588 | p. 7
(page number not for citation purposes)

JMIR AITao et al

Table 3. Prediction model performance on joint data with cross-validation. For completeness, all models are included.
Candidate machinelearning model

Precision orSpecificity, %Sensitivity, %PPVa, %

NPVb, %

F1-score

Balanced accuracy,%
Cost (US $)Random forest model weights
1

100

4.7

—c

99.5

—

52.35

659,016

200

77.2

78.2

1.77

99.85

0.034677.75d203,659.2

300

77.1

75.9

1.7

99.83

0.0332

76.5

220,284

500

79.1

74.1

1.82

99.83

0.0355d76.61

227,836.8

1000

79.2

71.7

1.77

99.81

0.034575.51

243,777.6

Decision tree model weights
1

99.3

10

—

99.53

—

54.66

623,973.6

200

73.9

75.9

1.51

99.82

0.0296d74.87d227,908.8

300

75

72.4

1.5

99.81

0.0294

73.66

249,696

500

59.7

83.5

1.09

99.85

0.0215

71.62

208,080

1000

68.6

74.1

1.22

99.8

0.024071.35

252,446.4

Logistic regression (L2) model weights
1

100

0

—

99.48

—

50

691,560

200

80.1

72.9

1.88

99.82

0.0366d

76.53

233,596.8

300

71.6

83.5

1.51

99.88

0.029677.6d

180,151.2

500

58.1

90.6

1.12

99.91

0.0221

74.36

162,964.8

1000

37.1

93.5

0.77

99.91

0.0152

65.33

191,757.6

691,560

Logistic regression (L1) model weights
1

100

0

—

99.48

—

50

200

80.1

72.9

1.88

99.82

0.036676.52

233,625.6
d

300

71.6

83.5

1.51

99.88

0.029677.59

180,151.2

500

58.1

90.6

1.12

99.91

0.0221

74.36

162,964.8

1000

45.7

88.2

0.84

99.87

0.016666.99

208,180.8

1

99.7

7

—

99.51

—

53.39

643,384.8

200

78.7

72.9

1.76

99.82

0.0343d75.84

236,800.8

300

71.1

84.1

1.5

99.88

0.029477.61d177,393.6

500

60.2

88.2

1.14

99.89

0.0225

74.21

174,456

1000

50

92.9

0.87

99.92

0.0172

68.96

177,429.6

e

SVM

a

PPV: positive predictive value.

b

NPV: negative predictive value.

c

Not available.

dBest model with respect to the specific metric.

e

SVM: support vector machines.

Compared with the best logistic regression and SVM models,
the random forest model with class weight 1:200 performs
slightly better in terms of both balanced accuracy and F1-score.
However, the random forest model with class weight 1:500 has
a better precision and F1-score. Combining the 2 metrics, we
decided to pick the random forest model with class weight 1:500
https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

for final deployment. The overall hyperparameters picked forthis model were a maximum tree depth of 6 and 100 total
estimators.
With this model in mind, we performed a more in-depth cost
analysis by varying the prediction threshold for the random
forest model and examining how these impact the expected
JMIR AI 2024 | vol. 3 | e48588 | p. 8
(page number not for citation purposes)

JMIR AI
medical costs and balanced accuracy. In Figures 5 and 6, weshow the results of the analysis for expected costs and balanced
accuracy, respectively. All values for this analysis werecomputed using leave-one-out cross-validation. As expected,

Tao et al
because the model was primarily chosen based on balanced
accuracy, this measure is maximized at a threshold of 0.5. On
the other hand, overall costs associated with model predictions
are maximized at a threshold of 0.3.

Figure 5. Plot showing a relationship between predictive threshold for the model and expected medical costs associated with treatment based on model
prediction. The y-axis is in thousands of US $ and the x-axis represents the threshold of prediction. All values were computed using leave-one-out
cross-validation and estimated costs from Niu et al [24].

Figure 6. Plot showing a relationship between predictive threshold for the model and balanced accuracy associated with treatment based on model
prediction. The y-axis is balanced accuracy and the x-axis represents the threshold of prediction. All values were computed using leave-one-out
cross-validation and estimated costs from the other cost analysis.

Risk Calculator
We operationalized our model by incorporating it into a risk
calculator that allows clinicians to compute how likely patientsare to be readmitted for hypertension-related factors. Figure 7
shows a screenshot of our calculator; clinicians are able to enter
9 numerical features in text fields and click 6 binary featuresusing a check box. The model for the calculator is deployed

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

using Python and Scikit-Learn [16] and is hosted on a public
website. The full code of the calculator model is publicly
available at the GitHub repository [19]. Based on the resultsfrom our previous analysis, in practice clinicians using this tool
may want to consider any likelihood above 30% as important
to consider when making treatment decisions to minimize
readmission risk and related medical costs.

JMIR AI 2024 | vol. 3 | e48588 | p. 9
(page number not for citation purposes)

JMIR AITao et al

Figure 7. Screenshot of calculator website.

Discussion
Principal Results
Our research indicates the importance of intrapartum andpostpartum blood pressure measurements in predicting
readmission. This is clinically important, as it suggests bloodpressure metrics before birth may be more important in guiding
postpartum hypertension treatment than previously
acknowledged. Current management of hypertension inpregnancy is based on expert opinion and has recommended
initiation of antihypertensive medication for postpartum systolicblood pressure greater than 150 mm Hg or diastolic bloodpressure greater than 100 mm Hg [4]. A recent study published
in AJOG MFM (American Journal of Obstetrics & Gynecology
Maternal-Fetal Medicine) suggests that lowering this thresholdto 140 mm Hg systolic or 90 mm Hg diastolic can increase
sensitivity in predicting postpartum readmission [10].
Regardless, given that systemic vascular resistance remains at
the pregnancy-associated lower value for about 2 days and thensubsequently increases to normal prepregnancy values by
postpartum day 3 to day 4, many women may be discharged
before the postpartum equilibration of blood pressure on
postpartum day 3 to day 4 and thus may be undertreated [20-23].
Using peak blood pressure values obtained during labor to aid
in decision-making may improve triaging and treatment ofhypertension after delivery, thus decreasing the risk of
postpartum readmission. Our research additionally indicatesthat blood pressure metrics themselves are more important inpredicting readmission than more typically used patient
demographics such as gestational age at delivery, maternal age,
BMI, laboratory data, or the administration of oral or intravenous
antihypertensive medication before discharge [11]. Perhaps theawareness of more severe diseases allows for more aggressive
treatment, management, and follow-up after the initial hospitalstay, thus decreasing readmission rates in this higher-risk group.

Comparison With Previous Work
Research evaluating postpartum readmission has useddescriptive statistics to describe demographic variables
https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

implicated in readmission. A nested case-control study publishedin the Journal of Perinatology in 2016 demonstrated noincreased risk of readmission by mode of delivery, severity ofpreeclampsia, fluid balance, use of magnesium sulfate, or lab
abnormalities but did find a decreased risk of readmission for
women discharged home on antihypertensive medication whencontrolled for age, race, and presence of chronic hypertension[24]. However, this study only included women withhypertension during their initial labor and delivery admission.
Given that 30% of women who experience hypertension-related
postpartum readmission do not have antecedent diagnoses ofhypertension, we furthered the previously published work by
Hirshberg et al [24] and included women without known
prepregnancy or pregnancy-induced hypertension in evaluating
for postpartum readmission in this study. Recently, an ML model
was published that evaluated factors predictive ofhypertension-related postpartum readmission [11]. Hoffman etal [11] evaluated 31 features in their model, similarly findingthat systolic blood pressure (specifically the moving average,or trend of the systolic blood pressure) was the most important
predictor of readmission. Our model identified biometric,
demographic, and obstetric variables easily identified in any
patient’s medical record. In addition, we investigated the useof specific antihypertensive medication, rather than using a drugscore that does not indicate which specific agents were used.
We used a cost-sensitive random forest method, allowing us toweigh the importance of particular observations and thus
penalize false negatives significantly higher than false positives.
Using our data and findings from this analysis, we created aclinical risk calculator [25]that predicts the likelihood of readmission based upon the key
clinical variables found to be most predictive ofhypertension-related postpartum readmission. Similar risk-based
calculators have previously been created and validated, including
the vaginal birth after cesarean calculator, commonly usedduring the antepartum period to guide counseling andmanagement of women with a previous cesarean section, andmore recently a calculator to estimate the risk of cesarean section
after an induction of labor with an unfavorable cervix [26,27].
JMIR AI 2024 | vol. 3 | e48588 | p. 10(page number not for citation purposes)

JMIR AIOur calculator applies our predictive model to any given patient
to predict the likelihood of readmission. While we do not define
when a patient should or should not be treated based on thelikelihood of readmission, we hope that better quantifying the
likelihood of readmission will allow for an improved discussion
between health care providers and their patients. However, basedon our cost analysis, it seems that if the calculator reports alikelihood of 30% or higher, clinicians may want to seriously
consider treatment options to reduce costs related to readmission.
Management options for women at higher risk of readmission
include earlier initiation of antihypertensive medication or closer
outpatient blood pressure surveillance with daily remote patientmonitoring or self-monitoring. These interventions would
hopefully lead to decreased health care costs by transitioning
to outpatient rather than inpatient care models. We recognizethat a balanced accuracy of 76.61% allows for error in ourmodel, thus health care providers must include this in theircounseling to optimize shared decision-making. Also, the 1.82%
precision that allows a number of false alarms should be noted.
However, since the cost of misidentifying a readmission is quite
high, the rate of false alarms might be necessary to provide
adequate care in the absence of other treatment options such ashome monitoring.

Strengths and Limitations
Strengths of our research include the development of a predictivemodel, different from the previously used descriptive models.
In addition, we had a large data set comprised from several
sources allowing for better validation and model development.
One limitation of our research is that the proposed predictivemodel is a random forest method, which is difficult to interpret.
Unlike logistic regression or a single decision tree, it is difficult
to extract exact thresholds for particular clinical measurements
to determine how they will impact the output of the random
forest. The importance plot of the random forest can be used tofind which features are most important for the model to makea prediction, but it cannot be used to determine particularTao et alprediction thresholds. This is of particular importance as thesethresholds will be key in establishing new treatment protocols.
In order to retrieve such thresholds, additional research needs
to be done in extracting an explainer model from our random
forest. Additional limitations include the use of ICD-9-CM(International Classification of Diseases, Ninth Revision,
Clinical Modification) and ICD-10 (International Statistical
Classification of Diseases, Tenth Revision) codes for diagnosis
of preexisting hypertension and HDP, which may have led toerrors in coding and underreporting. Our rate of hypertension
was 9%, which aligns with national data, but is lower than that
previously reported in Wisconsin (estimated at 22%). Finally,
our readmission rate was low at <1%. Given the large catchment
area of the institution used in our research, it is possible thatwomen who delivered at our hospital presented to their local
emergency department or provider with postpartum hypertension
and were thus not included in our data as a readmit. This study
is additionally limited by generalizability. Our patients camefrom a single, relatively homogenous, midwestern academic
institution. In order to apply these findings more broadly, ourpredictive model should be applied to a more diverse population.

Conclusions
Our research shows that blood pressure metrics during laborand post partum, in addition to obstetric and demographic
variables, are critical in creating a predictive model for
postpartum readmission. Predictive models like ours canimprove postpartum management, allowing practitioners tocharacterize women as low-risk and high-risk for readmission
and better individualize treatment. If we can better predict
readmission, we can better prevent readmission. By creating aclinical calculator to help guide postpartum hypertension
treatment, our goal is to decrease adverse maternal outcomesand prevent costly postpartum readmission. Future research will
involve validating this model and finding specific thresholdvalues at which treatment is to be initiated.

Acknowledgments
The authors would also like to thank Mr Dakota Dalton for his help with the deployment of the web-based calculator. The researchobtained financial support from the University of Wisconsin Department of Obstetrics & Gynecology Research and Development.
KKH was supported by the ObGyn Start-Up-UWF-SMPH Research and Development-AAB8581 award. No generative artificial
intelligence was used in the creation of the manuscript.

Data Availability
The data sets generated during or analyzed during this study are not publicly available as we do not standardly post or publish
data sets used for retrospective clinical research projects, and we also do not have Institutional Review Board’s approval to post
the data set publicly. However, the data set is available from the corresponding author on reasonable request.

Authors' Contributions
RGL wrote the initial draft of the manuscript and performed data curation through chart review. JT performed formal analysisof the data, validated the results, aided in the revisions, and worked on the software deployment of the online calculator. YM
supervised JT, created the core methodology for the analysis in the paper, and performed project administration. He also contributed
to writing the edited draft. OA assisted with methodology creation and writing of the original draft. KKH conceptualized thisproject idea, assisted with data collection, supervised RGL, and oversaw the research project.

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

JMIR AI 2024 | vol. 3 | e48588 | p. 11(page number not for citation purposes)

JMIR AITao et al

Conflicts of Interest
OA served as a paid consultant for Bristol Myers Squibb, Johnson & Johnson, and Exact Sciences outside the submitted work;
and is the owner of and principal scientist for Innovo Analytics LLC outside the submitted work.

Multimedia Appendix 1
Additional figures and tables, and supplementary analysis.
[DOCX File , 1894 KB-Multimedia Appendix 1]

References
1.
2.
3.
4.
5.

6.

7.
8.

9.

10.

11.

12.
13.
14.
15.
16.
17.

18.
19.
20.

21.
22.

Roberts CL, Algert CS, Morris JM, Ford JB, Henderson-Smart DJ. Hypertensive disorders in pregnancy: a population-based
study. Med J Aust. 2005;182(7):332-335. [doi: 10.5694/j.1326-5377.2005.tb06730.x] [Medline: 15804223]
Clapp MA, Little SE, Zheng J, Kaimal AJ, Robinson JN. Hospital-level variation in postpartum readmissions. JAMA.
2017;317(20):2128-2129. [FREE Full text] [doi: 10.1001/jama.2017.2830] [Medline: 28535223]
Clapp MA, Little SE, Zheng J, Robinson JN. A multi-state analysis of postpartum readmissions in the United States. Am
J Obstet Gynecol. 2016;215(1):113.e1-113.e10. [doi: 10.1016/j.ajog.2016.01.174] [Medline: 27829570]
NA. Hypertension in pregnancy. Report of the American college of obstetricians and gynecologists’ task force on hypertension
in pregnancy. Obstet Gynecol. 2013;122(5):1122-1131. [doi: 10.1097/01.AOG.0000437382.03963.88] [Medline: 24150027]
Parikh NI, Laria B, Nah G, Singhal M, Vittinghoff E, Vieten C, et al. Cardiovascular disease-related pregnancy complications
are associated with increased maternal levels and trajectories of cardiovascular disease biomarkers during and after pregnancy.
J Womens Health (Larchmt). 2020;29(10):1283-1291. [FREE Full text] [doi: 10.1089/jwh.2018.7560] [Medline: 31934809]
Parikh NI, Norberg M, Ingelsson E, Cnattingius S, Vasan RS, Domellöf M, et al. Association of pregnancy complications
and characteristics with future risk of elevated blood pressure: the västerbotten intervention program. Hypertension.
2017;69(3):475-483. [FREE Full text] [doi: 10.1161/HYPERTENSIONAHA.116.08121] [Medline: 28137991]
Kaaja RJ, Greer IA. Manifestations of chronic disease during pregnancy. JAMA. 2005;294(21):2751-2757. [doi:10.1001/jama.294.21.2751] [Medline: 16333011]
Fraser A, Nelson SM, Macdonald-Wallis C, Cherry L, Butler E, Sattar N, et al. Associations of pregnancy complications
with calculated cardiovascular disease risk and cardiovascular risk factors in middle age: the Avon Longitudinal Study of
Parents and Children. Circulation. 2012;125(11):1367-1380. [FREE Full text] [doi:10.1161/CIRCULATIONAHA.111.044784] [Medline: 22344039]
Mogos MF, Salemi JL, Spooner KK, McFarlin BL, Salihu HH. Hypertensive disorders of pregnancy and postpartum
readmission in the United States: national surveillance of the revolving door. J Hypertens. 2018;36(3):608-618. [doi:10.1097/HJH.0000000000001594] [Medline: 29045342]
Mukhtarova N, Alagoz O, Chen YH, Hoppe K. Evaluation of different blood pressure assessment strategies and cutoff
values to predict postpartum hypertension-related readmissions: a retrospective cohort study. Am J Obstet Gynecol MFM.
2021;3(1):100252. [doi: 10.1016/j.ajogmf.2020.100252] [Medline: 33451628]
Hoffman MK, Ma N, Roberts A. A machine learning algorithm for predicting maternal readmission for hypertensive
disorders of pregnancy. Am J Obstet Gynecol MFM. 2021;3(1):100250. [doi: 10.1016/j.ajogmf.2020.100250] [Medline:
33451620]
PeriData. URL: https://www.peridata.net/peridata/security/access.htm [accessed 2024-08-14]
McKinney W. Data structures for statistical computing in python. 2010. Presented at: Proceedings of the 9th Python in
Science Conference; 2010 Jun 28:51-56; Austin, Texas. [doi: 10.25080/majora-92bf1922-00a]
Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed.
New York. Springer-Verlag; 2009:9780387848587.
Tharwat A. Classification assessment methods. ACI. 2020;17(1):168-192. [doi: 10.1016/j.aci.2018.08.003]
Pedregosa F, Varoquaux G, Gramfort A, Michel V. Scikit-learn: machine learning in python. Journal of Machine Learning
Research. 2011;12:2825-2830.
Niu B, Mukhtarova N, Alagoz O, Hoppe K. Cost-effectiveness of telehealth with remote patient monitoring for postpartum
hypertension. J Matern Fetal Neonatal Med. 2022;35(25):7555-7561. [doi: 10.1080/14767058.2021.1956456] [Medline:
34470135]
Hypertension prediction. URL: https://github.com/jtao34/hypertension-prediction [accessed 2024-08-14]
Hypertension calculator. URL: https://github.com/uwobgyn/HypertensionCalculator [accessed 2024-08-14]
Hibbard JU, Schroff SG, Cunningham FG. Cardiovascular alterations in normal and preeclamptic pregnancy. In: Taylor
RN, Roberts JM, Cunningham FG, editors. Chesley's Hypertensive Disorders in Pregnancy (Fourth Edition). US. Academic
Press; 2014:291-313.
Christianson RE. Studies on blood pressure during pregnancy. I. influence of parity and age. Am J Obstet Gynecol.
1976;125(4):509-513. [doi: 10.1016/0002-9378(76)90367-7] [Medline: 984086]
Cunningham FG, Leveno KJ, Bloom SL, Spong CY, Dashe JS, Hoffman BL, et al. Obstetrics W, editor. The Puerperium.
US. McGraw-Hill Education; 2014:668-681.

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

JMIR AI 2024 | vol. 3 | e48588 | p. 12(page number not for citation purposes)

JMIR AI
23.
24.

25.
26.

27.

Tao et al

Walters BN, Thompson ME, Lee A, de Swiet M. Blood pressure in the puerperium. Clin Sci (Lond). 1986;71(5):589-594.
[doi: 10.1042/cs0710589] [Medline: 3769407]
Hirshberg A, Levine LD, Srinivas SK. Clinical factors associated with readmission for postpartum hypertension in womenwith pregnancy-related hypertension: a nested case control study. J Perinatol. 2016;36(5):405-409. [FREE Full text] [doi:10.1038/jp.2015.209] [Medline: 26765549]
Staying Healthy After Childbirth. Department of Obstetrics and Gynecology; 2023. URL: https://www.obgyn.wisc.edu/
stac/ObGyn.HypertensionCalculator [accessed 2024-07-01]
Grobman WA, Lai Y, Landon MB, Spong CY, Leveno KJ, Rouse DJ, et al. National Institute of Child HealthHuman
Development (NICHD) Maternal-Fetal Medicine Units Network (MFMU). Development of a nomogram for prediction ofvaginal birth after cesarean delivery. Obstet Gynecol. 2007;109(4):806-812. [doi: 10.1097/01.AOG.0000259312.36053.02][Medline: 17400840]
Levine LD, Downes KL, Parry S, Elovitz MA, Sammel MD, Srinivas SK. A validated calculator to estimate risk of cesarean
after an induction of labor with an unfavorable cervix. Am J Obstet Gynecol. 2018;218(2):254.e1-254.e7. [FREE Full text][doi: 10.1016/j.ajog.2017.11.603] [Medline: 29224730]

Abbreviations
AJOG MFM: American Journal of Obstetrics & Gynecology Maternal-Fetal Medicine
EMR: electronic medical record
HDP: hypertensive disorders of pregnancy
ICD-9-CM: International Classification of Diseases, Ninth Revision, Clinical Modification
ICD-10: International Statistical Classification of Diseases, Tenth Revision
ML: machine learning
ROC: receiver operating characteristic
SVM: support vector machines

Edited by K El Emam, B Malin; submitted 28.04.23; peer-reviewed by J Luo, S Sarejloo, T Wang, T Kotani; comments to author
02.01.24; revised version received 07.02.24; accepted 23.06.24; published 13.09.24
Please cite as:
Tao J, Larson RG, Mintz Y, Alagoz O, Hoppe KK
Predictive Modeling of Hypertension-Related Postpartum Readmission: Retrospective Cohort Analysis
JMIR AI 2024;3:e48588
URL: https://ai.jmir.org/2024/1/e48588
doi: 10.2196/48588
PMID:©Jinxin Tao, Ramsey G Larson, Yonatan Mintz, Oguzhan Alagoz, Kara K Hoppe. Originally published in JMIR AI(https://ai.jmir.org), 13.09.2024. This is an open-access article distributed under the terms of the Creative Commons Attribution
License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any
medium, provided the original work, first published in JMIR AI, is properly cited. The complete bibliographic information, alink to the original publication on https://www.ai.jmir.org/, as well as this copyright and license information must be included.

https://ai.jmir.org/2024/1/e48588

XSL• FO
RenderX

JMIR AI 2024 | vol. 3 | e48588 | p. 13(page number not for citation purposes)