Using machine-learning algorithms to improve imputation in the Medical Expenditure Panel Survey

Chandler McClellan, Emily M. Mitchell, Jerrod Anderson, and Samuel H. Zuvekas

Federal Committee on Statistical Methodology Annual Meeting

2021

1

Disclaimer

The views expressed in this presentation are those of the author and no official endorsement by the Department of Health and Human Services or the Agency for Healthcare Research and Quality is intended or should be inferred.

2

Data on frequency and cost of health services

Nationally-representative

Civilian, non-institutionalized U.S. population

Annually, since 1996

Public-use files

Medical Expenditure Panel Survey (MEPS)

3

“MEPS data are the number one single source of data for papers published in Health Affairs.”

Civilian (e.g. not military), non-institutionalized (e.g. no nursing homes, no jails) population.

3

Medical Expenditure Panel Survey (MEPS)

Household Component (HC)

Medical Provider Component (MPC)

4

Utilization

Expenditures

Expenditures

Event Types: ER, Inpatient, Outpatient, Medical Visits

Sources of Payments: Total Expenditure, Out-of-pocket, Medicare, Medicaid, Private Insurance, VA, TriCare/CHAMPUS, State and Local, Worker’s Comp., Other 

We start by interviewing each household, to ask about recent medical events (utilization). Then, with their permission, we go to their providers to get detailed information about the event, particularly expenditure data, (e.g. total charge, how much was paid by the patient, how much was paid by insurance.)

MPC is considered the gold standard, but we also collect similar expenditure data from HC.

Which providers we contact (All ER/HS, etc.)

4

Dealing with Missing Data

Households don’t remember

5

Trends in the Proportion of Medical Events with Imputed Total Expenditure Values in the MEPS 

Emergency Room

Hospitalization

Medical Visit

Outpatient

Medical Expenditure Panel Survey (MEPS)

6

Household Component (HC)

Imputation (predictive mean matching)

Medical Provider Component (MPC)

1

2

3

MPC is considered the gold standard, so we use that data whenever possible. 

If MPC is not available, then we use the data that the HC provided.

If neither are available, then we impute (using predictive mean matching)

6

Predictive Mean Matching

Predictive Mean Matching

7
| Out-of-pocket payments | Third-party payments |
|------------------------|----------------------|
|                        |                      |
| X                      |                      |
| X                      | X                    |
|                        |                      |
| X                      |                      |
| X                      | X                    |
| Complete MPC data |
|-------------------|
| X                 |
| X                 |
| X                 |
|                   |
|                   |
|                   |

Donors

Recipients

(not included in imputation)

All events with complete MPC data get assigned MPC data

Otherwise, events with complete HC data get assigned HC data, and don’t enter into the imputation 

Otherwise, the event is imputed, and is a recipient

7

Predictive Mean Matching

8

(donors)

(recipients)
| 1. | Model  |
|----|--------|
| 2. | Fit    |
| 3. | Match* |
| 4. | Impute |

*Within groups with similar sources 

of payment available (XGROUPS)

8

Predictive Mean Matching - Current

9
|                        | HC data | MPC data |
|------------------------|---------|----------|
| IMPFLAG = 1            | X       |          |
| IMPFLAG = 2            | X       | X        |
| IMPFLAG = 5(capitated) | X       | X        |
| IMPFLAG = 3/4          |         |          |

Imputation

Not included in imputation

recipients

donors

1. Fit MLR model

2. Predict 
| X | X |
|---|---|
| X | X |
|  |  |
|--|--|

donors

recipients

3. Match within XGROUP

4. Impute

recipients

donors
| SUMPAYX | TLCHRG |
|---------|--------|
|         |        |
| SUMPAYX | TLCHRG |
|---------|--------|
|         |        |

For each event type.

Different predictors for each MLR

9

Can we do better?

Modeling algorithm/fit

Currently OLS

Machine learning offers alternatives for prediction

Neural Networks

Random Forests

Gradient Boosting

Stacked Ensemble

Training completed with H2O.ai

Matching

Currently match on single target – Total Expenditure

Alternatively predict and match on Total Expenditure and Payment Sources

Better alignment on payment sources and eliminate need for XGROUPS

10

Predictive Mean Matching

11

(donors)

(recipients)
| 1. | Model  |
|----|--------|
| 2. | Fit    |
| 3. | Match  |
| 4. | Impute |

11

PMM – testing alternatives

12
|                        | HC data | MPC data |
|------------------------|---------|----------|
| IMPFLAG = 1            | X       |          |
| IMPFLAG = 2            | X       | X        |
| IMPFLAG = 5(capitated) | X       | X        |
| IMPFLAG = 3/4          |         |          |
| X | X |
|---|---|
| X | X |
|  |  |
|--|--|

New donors

New recipients

3. Match Vector

4. Impute
| SUMPAYX | TLCHRG |
|---------|--------|
|         |        |

Train

‘new donors’

Test 

‘new recipients’

1. Fit ML algorithm

2. Predict 

New donors

New recipients

EVALUATE R^2

Compare Ypred to Ytrue

EVALUATE R^2

Compare Yimp to Ytrue
| SUMPAYX | TLCHRG |
|---------|--------|
|         |        |

“Kaggle” competition – give all algorithms same data set, same covariates (a.k.a. features), same outcome (a.k.a. label, target), same matching process.

Using data only from previously identified donors 

Split data into training data (‘donors’) and test data (‘recipients’)

Produce imputations for all event types

Estimate process on single years of data and pooled years of data

Compare fit

RMSE, MAE, R2, etc.

Compare matches

Compare expenditure estimates

12

Accuracy Evaluation

Prediction R^2

Imputation R^2

13

Prediction Accuracy – Single Target

14
|                        | Event Type     |                 |            |               |
|------------------------|----------------|-----------------|------------|---------------|
| Algorithm              | Emergency Room | Hospitalization | Outpatient | Medical Visit |
| Ordinary Least Squares | 0.27           | 0.365           | 0.264      | 0.086         |
| Stacked Ensemble       | 0.358          | 0.46            | 0.44       | 0.352         |
| Gradient Boosting      | 0.318          | 0.413           | 0.428      | 0.333         |
| Random Forest          | 0.318          | 0.411           | 0.396      | 0.341         |
| Extreme Random Forest  | 0.32           | 0.415           | 0.408      | 0.333         |
| Deep Neural Net        | 0.283          | 0.424           | 0.375      | 0.219         |

Imputation Accuracy – Single Target

15
|                        | Event Type     |                 |            |               |
|------------------------|----------------|-----------------|------------|---------------|
| Algorithm              | Emergency Room | Hospitalization | Outpatient | Medical Visit |
| Ordinary Least Squares | -0.438         | 0.071           | -0.127     | -0.298        |
| Stacked Ensemble       | 0.25           | 0.408           | 0.069      | 0.069         |
| Gradient Boosting      | 0.06           | 0.345           | 0.038      | 0.019         |
| Random Forest          | 0.263          | 0.396           | 0.014      | 0.052         |
| Extreme Random Forest  | 0.172          | 0.393           | 0.015      | 0.047         |
| Deep Neural Net        | -0.062         | 0.37            | -0.173     | -0.195        |

Stacked Ensemble Prediction Accuracy – Payment Source

16
|                       | Event Type     |                 |            |               |
|-----------------------|----------------|-----------------|------------|---------------|
| Expenditure Type      | Emergency Room | Hospitalization | Outpatient | Medical Visit |
| Out of Pocket         | 0.714          | 0.503           | 0.643      | 0.717         |
| Medicare              | 0.754          | 0.758           | 0.597      | 0.64          |
| Medicaid              | 0.658          | 0.711           | 0.659      | 0.716         |
| Private               | 0.649          | 0.698           | 0.656      | 0.63          |
| VA                    | 0.991          | 0.634           | 0.915      | 0.802         |
| Tricare/CHAMPUS       | 0.675          | 0.739           | 0.486      | 0.848         |
| Other State and Local | -0.616         | 0.997           | 0.484      | 0.813         |
| Worker's Comp         | 0.709          | -               | 0.528      | 0.716         |
| Other                 | 0.63           | -0.044          | 0.655      | 0.739         |
| Total Expenditure     | 0.254          | 0.452           | 0.41       | 0.332         |

Average Imputation Accuracy across Payment Sources of Vector Matching
|             | Emergency Room | Hospitalization | Outpatient | Medical Visit |
|-------------|----------------|-----------------|------------|---------------|
| OLS- Single | -3.771         | -1.995          | -0.233     | -0.220        |
| SE - Single | -2.110         | -2.029          | -0.208     | -0.064        |
| SE - Vector | 0.613          | 0.427           | 0.386      | 0.615         |

17

Imputation Accuracy – Vector Matching

18
|                       | Event Type     |             |             |                 |             |             |             |             |             |               |             |             |
|-----------------------|----------------|-------------|-------------|-----------------|-------------|-------------|-------------|-------------|-------------|---------------|-------------|-------------|
| Source of Payment     | Emergency Room |             |             | Hospitalization |             |             | Outpatient  |             |             | Medical Visit |             |             |
|                       | OLS- Single    | SE - Single | SE - Vector | OLS- Single     | SE - Single | SE - Vector | OLS- Single | SE - Single | SE - Vector | OLS- Single   | SE - Single | SE - Vector |
| Out of Pocket         | -0.913         | -1.39       | 0.554       | -17.9           | -17.8       | 0.224       | -0.427      | -0.312      | 0.522       | -0.149        | 0.014       | 0.66        |
| Medicare              | -1.36          | -0.001      | 0.665       | 0.103           | 0.283       | 0.759       | -1.16       | -1.61       | 0.31        | -0.699        | -0.2        | 0.505       |
| Medicaid              | -0.129         | -0.076      | 0.594       | -0.151          | -0.251      | 0.708       | -0.086      | -0.357      | 0.483       | -0.233        | -0.12       | 0.615       |
| Private               | -1.71          | 0.151       | 0.556       | -0.134          | -0.14       | 0.655       | -0.568      | 0.07        | 0.49        | -0.615        | -0.322      | 0.437       |
| Veteran’s Admin.      | -0.082         | 0.325       | 0.81        | -0.006          | -0.006      | 0.145       | 0.114       | 0.127       | 0.166       | 0.316         | 0.74        | 0.792       |
| Tricare/CHAMPUS       | -4.38          | -10.6       | 0.716       | 0.084           | -0.03       | 0.727       | 0.02        | 0.016       | 0.446       | -0.672        | -1.43       | 0.734       |
| Other State and Local | -28.6          | -9.46       | 0.432       | -0.014          | -0.003      | -0.003      | 0.019       | 0.019       | 0.133       | 0.771         | 0.515       | 0.931       |
| Worker's Comp         | -0.066         | -0.007      | 0.881       | -               | -           | -           | -0.021      | -0.001      | 0.523       | 0.18          | 0.186       | 0.684       |
| Other                 | -0.04          | -0.293      | 0.71        | -0.01           | -0.723      | 0.205       | -0.096      | -0.105      | 0.626       | -0.797        | -0.088      | 0.723       |
| Total Expenditure     | -0.438         | 0.25        | 0.208       | 0.071           | 0.408       | 0.438       | -0.127      | 0.069       | 0.162       | -0.298        | 0.069       | 0.072       |

Matching – Proportion of Imputed with exact Sources of Payment Pattern as True
|    | XGROUP | Vector |
|----|--------|--------|
| ER | 60.81  | 98.27  |
| HS | 59.47  | 95.00  |
| ME | 64.72  | 97.08  |
| OP | 71.90  | 99.00  |

19

Direct Prediction Vs. Imputation

20

Predicted

Imputed

Conclusions

ML algorithms produce better predictions than OLS

Predictions tend to compress distributions.

Overestimate at low end, Underestimate at high end

Matching addresses this

Better matching with ML as well.

Even better matching by expanding number of targets

21

# image3.png

# image4.png

# image5.png

# image6.png

# image7.png

# image8.png

# image70.png

# image80.png

# image9.png

# image10.png

# image11.png

# image12.png

# image130.png

# image14.png

# image15.png

# image16.png

# image17.png

# image13.png

# image18.png

# image19.png

# image20.png

# image1.jpg

# image2.jpg