Assessment of the effects of COVID-19 on data collected by the Medical Expenditure Panel Survey (MEPS)

Alisha Creel, Ralph DiGaetano, Hanyu Sun, Alexis Kokoska, David Cantor

1

Hello everyone. Thanks to Brad and Rick for their overview of the changes that MEPS has experienced during the COVID-19 pandemic. Today I am going to review work that Ralph, Hanyu, Alexis, David and I did earlier in 2021 to assess the effects of these changes on the data collected for MEPS.

1

Disclaimer

The statements in this presentation are those of the presenter and do not necessarily represent the views of the Agency for Healthcare Research and Quality (AHRQ) or the Department of Health and Human Services (DHHS).

2

Our standard disclaimer that the statements in this presentation are our own and do not represent the views of AHRQ or DHHS.

2

Agenda

Context and Overall Findings

Use of Records

Show Card Items

Key Survey Estimates: Health Care Utilization and Health Insurance

Future Directions

3

For today’s agenda, [X]upfront I’m going to give you just a little context and summarize the findings, so you know where we’re headed. Then I’ll give you more detail about how we got to each of the findings. [X]We looked at respondent use of records, [X]items where we ask respondents to use show cards, [X]and key survey estimates, which for MEPS are around health care utilization and health insurance status. [X]At the end I will talk about future analysis and ongoing work on the survey to accommodate needs around COVID-19.

3

Context

MEPS key estimates: Health care utilization and health insurance

Important for quality

Records

Show cards

Changes due to COVID-19

Shift from in person to telephone

Extension of exiting panel

4

So first some brief context. [X]MEPS’s key estimates are focused on health care utilization and health insurance status. That’s our main focus when it comes to data quality.

[X]I’ll talk more about this later on, but a couple of tools we use to improve data quality on MEPS is [X]one, to encourage respondents to use records to help them remember their health care events and, [X]two, to use show cards to help trigger respondents’ memories and help them respond to items that have a lot of potential response options.

[X]During COVID we made two major changes that we wanted to assess as far as data quality. [X]The first one was the shift from in person to telephone interviewing. [X]The second was extending the participation of the panel that was originally supposed to exit the study in 2020. We had them do their spring interview as the normal exit interview, reporting events through the end of 2019, but then in the fall we invited them back and had them report on events for all of 2020 to the interview date.

4

Overall Findings

Households switched to telephone less likely to use records than those who did interview in person

No strong differences in response distributions for items using show cards

For health insurance status, no evident impact from mode change or COVID on estimates

Mode switch did not seem to impact event reporting: 2019 did not differ by mode, 2020 tracked with decreases due to COVID

Extended panel reported fewer events than expected

5

So here are our findings. [X]First, households that switched to having their spring 2020 interview done by phone were less likely to use records than those who did the spring 2020 interview in person.

[X]Next, we found no strong differences between modes in the response distributions for items where we use show cards.

[X]We did not find any impact of either mode or the COVID pandemic on health insurance status

[X]Furthermore, the mode switch did not seem to impact event reporting, but we did see decreases in health care events that we believe were related to the COVID pandemic.

[X]And finally the extended panel reported fewer events than expected for the bulk of 2020.

5

**Angie Kistler (AK) -**I would say the covid pandemic - because just saying COVID - could people interpret as the condition?

**Angie Kistler (AK) -**I don't feel like the slide immediately  prior to this fully explained the extended panel - so this last statement is a little confusing.  Even if the presentation prior to this explains it, you may want to build in a little more context for those just tuning in.

**Alisha Creel (AC) -**Agree on insertion of pandemic, added a little more context re: extended panel (I get into more detail later).

Use of Records

Encourage respondents to use records to get most accurate information about health events

Calendar, bills, insurance statements, prescription bottles, etc.

Data quality concerns: In-person vs. telephone

Mode: Compared respondents with in person interviews in both fall 2019 and spring 2020 with those who switched to phone in spring 2020

6
| Group             | Fall 2019           | Spring 2020         |
|-------------------|---------------------|---------------------|
| In Person Group   | In Person Interview | In Person Interview |
| Mode Switch Group | In Person Interview | Phone Interview     |

So let’s talk about the use of records. The MEPS Household Component is primarily focused on collecting information about health care events from respondents. [X]To make sure we get the best data, we encourage respondents to use records as they respond to our questions. [X]These records can include their calendar, bills, insurance statements, prescription bottles, scraps of paper, anything that will provide information about the dates of events, the providers, and so on. During in person interviews, interviewers can encourage respondents to gather up their records and even help respondents organize and extract information from the records. So records are important for data quality, and we lead off the section that collects health care events by asking respondents what records they have available and collecting the events from those first.

[X]The shift to telephone interviewing was of concern for records use, because we no longer had the interviewer there to encourage records use or help the respondent sift through them. So we wanted to look at records use by mode. 

[X]So to look at mode, we looked at the two panels that had a fall 2019 interview and a spring 2020 interview. We used a quasi-experimental design to compare the change in records use for respondents who had in-person in both fall 2019 and spring 2020 with those who switched to phone in spring 2020. We used a difference-in-differences analysis to control for the baseline differences between respondents. So we’re looking at the change over time within households and comparing that change across these two groups.

6

Use of Records by Mode Groups
| Variable                                       | In-PersonPercent (SE) | Mode SwitchPercent (SE) | Test Statistics   |
|------------------------------------------------|-----------------------|-------------------------|-------------------|
| Panel 23                                       |                       |                         |                   |
| Records in fall 2019,No records in spring 2020 | 13.6                  | 23.8                    | χ2 (2) = 120.568  |
| No change between fall 2019 and spring 2020    | 77.4                  | 66.1                    | p-value = 0.002   |
| No records in fall 2019,Records in spring 2020 | 9.0                   | 10.2                    |                   |
| Panel 24                                       |                       |                         |                   |
| Records in fall 2019,No records in spring 2020 | 10.6                  | 17.6                    | χ2 (2) = 81.542   |
| No change between fall 2019 and spring 2020    | 74.3                  | 68.9                    | p-value = 0.002   |
| No records in fall 2019,Records in spring 2020 | 15.0                  | 13.5                    |                   |

7

This table shows the results for the two panels that had interviews at both time points.

We will focus on the change from having records in fall 2019 to not having records in spring 2020. [X]For panel 23, 13.6% of in person respondents went from having records to not, compared to 23.8% of mode switch respondents. [X]Likewise, for panel 24, 10.6% of in person respondents went from having records to not, compared to 17.6% of mode switch respondents. This would seem to indicate the possibility that the switch to telephone has a negative impact on records use. Mode switch respondents were by definition later respondents in the spring 2020 round, so it’s also possible these were less cooperative respondents than the folks who had already completed an in person interview earlier in the spring. But they had used records in the fall.

7

Items with Show Cards

Provide response options, examples when list is long

Data quality concerns: In person vs. telephone

Recency effects

8

Next up, items with show cards. [X]MEPS has a lot of complex questions where we ask respondents to look at show cards with all the possible response options or examples we want them to consider. In the typical in-person administration, interviewers give respondents a binder of show cards and direct respondents to each card. Interviewers only read the show card aloud if respondents can’t. Otherwise there is a kind of social pressure to at least appear to comply with the interviewers’ request. When MEPS switched to the telephone mode, interviewers asked respondents to go to a website to access the show cards for use during the MEPS interview. Interviewers read the show card aloud to the respondent if and when it was clear the respondent wasn’t accessing the show cards online.

[X]There are a few data quality concerns we would have with the interviewer having to read the show cards. [X]Where there is a long list of response options, we would be concerned about recency effects, where the response options at the end of a long list presented orally are more likely to be selected because they’re last and the respondent remembers them better than the earlier response options. 

8

**Angie Kistler (AK) -**You have yes/no items here and then an example on slide 11 - do you want to at least mention it here?

**Alisha Creel (AC) -**I reorganized to come back to this slide in between, thanks

Show Card Example: Health Event Type (Single response)

9

Here’s an example of a single response item with a long list of possible responses. A respondent has told us they had a health care event and we are asking them at what type of place did they receive the care. This is a lot to read out loud, and the respondent might forget about the first few options by the time they hear the whole list. So we might expect to see a recency bias in the response distribution.

9

Items with Show Cards

Provide response options, examples when list is long

Data quality concerns: In person vs. telephone

Recency effects

Multiple response and yes/no items

10

Our other concern would be multiple response or yes/no items, where we might be concerned that without seeing the examples on the show card, respondents would be less likely to endorse response options or answer yes.

10

Show Card Example: Services received (Multiple response)

11

Here’s an example of a multiple response item. We want to know during a doctor visit which of these services, if any, the person received. Here we would be concerned that respondents might not understand the interviewer reading a long list of examples over the telephone in the same way as they would comprehend reading the list to themselves from a show card in an in-person interview. Maybe they would endorse fewer of these over the phone than in person.

11

Show Card Example: Long Term Medical Equipment Purchased (Y/N) 

12

This is a yes/no question about whether the person purchased any long term medical equipment. The show cards lists a ton of examples of what this could include. Here we would be concerned that the respondent would just get lost listening to this whole show card and might miss a relevant example. If so, then there might be fewer “yes” responses from these respondents since not all of the examples were taken into consideration when answering the question.

12

Items with Show Cards

Provide response options, examples when list is long

Data quality concerns: In person vs. telephone

Recency effects

Yes/no items

Compared in person vs. mode switch groups

Variety of items with show cards

No strong evidence of mode effects

Differences often reflected differences between groups

Don’t know how often telephone respondents used web show cards

13

To assess these data quality concerns, we compared our in person and mode switch groups in their response distributions on a variety of items with show cards. [X]Bottom line is that we didn’t see many differences between the groups in spring 2020. [X]In some cases where we did, the groups actually differed in the same way in their fall 2019 interviews when everyone was interviewed in person, reflecting differences not due to mode. [X]We didn’t track when respondents purportedly used the web show cards versus when interviewers had to read the show card, however, and based on discussions with interviewers, most respondents were pretty cooperative about going online and using the show cards. So it may be that any impact of reading the cards out loud would be diminished.

13

Key Survey Estimates: Heath Care Utilization and Health Insurance Status

Key focus for data quality

Mode shift concurrent with change in health care use

Effect on 2019 events vs. 2020 events

Extended panel

Expected to exit: Spring 2020 interview reference period ended 12/31/2019

Fall 2020 interview reference period started 1/1/2020 instead of date of spring interview

Health insurance: Less concern than health events

14

MEPS’s key estimates are focused on health care expenditures, but to get those we need to know about health care utilization or the health events that people experienced. Health insurance status is also important. These are our main focus when it comes to data quality.

[X]One thing to note about the shift in mode from in person to telephone is that it occurred at the same time as a known decrease in the use of elective health care. This makes looking at mode effects pretty difficult since we would expect telephone respondents to report fewer health care events once the COVID-related lockdowns started. [X]But luckily for us, the spring interview asks exiting and continuing respondents about health care events from the end of 2019, and continuing respondents about health care events from 2020. So we could see if there was an impact of mode on the 2019 events, where the COVID pandemic should not have had an impact.

[X]So you’ll recall the other big change we made due to covid was to extend the participation of the panel that we had expected to exit in spring 2020. We did this to boost the number of cases that will be in the data file, since we expected to have lower response rates on the phone for the other panels. Because we expected these folks to be done, in the spring 2020 interview their reference period for reporting health events ended December 31, 2019. [X]So when we decided to extend them, when we interviewed them in the fall, we had to cover all of 2020 to that point, all the way back to January 1. Normally the fall reference period on MEPS goes back to the date of the spring interview, so let’s say it’s September and I last talked to you in March, we only have to cover 6 months. But for these extended panel folks, now we’re covering 9 months. Furthermore, even the best most cooperative respondents who were using records in their interviews may have stopped keeping them or keeping them so well organized after they thought they were done with MEPS. So we were worried about data quality for this panel.

[X]Finally, we also looked at health insurance status, although there weren’t as many a priori concerns to point to. We could have anticipated some changes in health insurance status due to covid-related employment shifts that might have overlapped with the shift in mode, but we weren’t as concerned about mode effects. We would expect people to be equally informed or clueless about their health insurance in either mode.

14

Key Survey Estimates: Heath Care Utilization and Health Insurance Status

Compared full year estimates for 2019 to 2018 and 2017

No evidence for drop in 2019 utilization estimates, were more often higher

Estimates of percent uninsured were stable throughout 2019, mirrored prior years

Compared 2020 across panels

Entering panel somewhat higher number of events than continuing panel, extended panel the lowest

Health insurance status similar across three panels

15

We looked at the impact of COVID on health care utilization and health insurance status in a number of different ways. [X]First, we simply compared the 2019 estimates to the 2018 and 2017 estimates to see if 2019 estimates were notably different. [X]For events, where there were differences it was more often that 2019 showed more events. [X]For health insurance, we looked at monthly estimates of the percent uninsured, and the estimates were stable throughout 2019, mirroring 2017 and 2018.

[X]For 2020, we compared events per person across panels. For the 2020 data, when we did the analysis we only had data collected through fall 2020, or about two–thirds of the year. Because it hadn’t gone through full year data processing yet, we used makeshift weights and there was little data editing. So with some caveats, [X]we found that the new or entering panel households, Panel 25, reported somewhat more events than the continuing panel, Panel 24. Panel 23, the extended panel that was supposed to have exited, reported the fewest events. I’ll show that on the next slide. [X]As far as health insurance status the three panels looked the same.

15

2020 Health Care Utilization
|                     | P25 R1/2 | P24 R3/4 | P23 R6 | P25 v P24 | P24 v P23 |
|---------------------|----------|----------|--------|-----------|-----------|
| Overall             | Mean     | Mean     | Mean   | p value   | p value   |
| Office Provider     | 4.36     | 3.98     | 3.29   | 0.07      | <.01   |
| Office Physician    | 1.99     | 1.91     | 1.65   | 0.20      | <.01   |
| Outp Provider       | 0.53     | 0.44     | 0.39   | 0.08      | 0.17      |
| Outp Physician      | 0.25     | 0.19     | 0.17   | 0.07      | 0.19      |
| Emergency Room      | 0.13     | 0.12     | 0.10   | 0.16      | <.01   |
| Hospital Discharges | 0.05     | 0.05     | 0.06   | 0.96      | 0.72      |
| Dental Care         | 0.58     | 0.53     | 0.46   | 0.01      | <.01   |

16

This table shows the event results. Provider in this table means non-physician providers like physician assistants, nurses and so on. When comparing Panels 25 and 24, Panel 25, the new panel, reported more dental visits than Panel 24, the continuing panel. While other event types did not reach statistical significance, in general Panel 25 reported more events than Panel 24.

Many more differences were found to be significantly different Panel 24 to Panel 23 estimates. The extended panel reported fewer Office Based Provider and Physician Visits as well as ER visits and Dental Visits.

16

**Angie Kistler (AK) -**what's the difference between an office "provider" and an office "physician"?  is provider=facility? Will the average viewer of the presentation pick up on/care about this?

**Alisha Creel (AC) -**I added a line defining (I think that's what we decided these were).

2020 Health Care Utilization
|                     | P25 R1/2 | P24 R3/4 | P23 R6 | P25 v P24 | P24 v P23 |
|---------------------|----------|----------|--------|-----------|-----------|
| Overall             | Mean     | Mean     | Mean   | p value   | p value   |
| Office Provider     | 4.36     | 3.98     | 3.29   | 0.07      | <.01   |
| Office Physician    | 1.99     | 1.91     | 1.65   | 0.20      | <.01   |
| Outp Provider       | 0.53     | 0.44     | 0.39   | 0.08      | 0.17      |
| Outp Physician      | 0.25     | 0.19     | 0.17   | 0.07      | 0.19      |
| Emergency Room      | 0.13     | 0.12     | 0.10   | 0.16      | <.01   |
| Hospital Discharges | 0.05     | 0.05     | 0.06   | 0.96      | 0.72      |
| Dental Care         | 0.58     | 0.53     | 0.46   | 0.01      | <.01   |

17

This table shows these results. When comparing Panels 25 and 24, Panel 25, the new panel, reported more dental visits than Panel 24, the continuing panel. While other event types did not reach statistical significance, in general Panel 25 reported more events than Panel 24.

17

2020 Health Care Utilization
|                     | P25 R1/2 | P24 R3/4 | P23 R6 | P25 v P24 | P24 v P23 |
|---------------------|----------|----------|--------|-----------|-----------|
| Overall             | Mean     | Mean     | Mean   | p value   | p value   |
| Office Provider     | 4.36     | 3.98     | 3.29   | 0.07      | <.01   |
| Office Physician    | 1.99     | 1.91     | 1.65   | 0.20      | <.01   |
| Outp Provider       | 0.53     | 0.44     | 0.39   | 0.08      | 0.17      |
| Outp Physician      | 0.25     | 0.19     | 0.17   | 0.07      | 0.19      |
| Emergency Room      | 0.13     | 0.12     | 0.10   | 0.16      | <.01   |
| Hospital Discharges | 0.05     | 0.05     | 0.06   | 0.96      | 0.72      |
| Dental Care         | 0.58     | 0.53     | 0.46   | 0.01      | <.01   |

18

Many more differences were found to be significantly different Panel 24 to Panel 23 estimates. The extended panel reported fewer Office Based Provider and Physician Visits as well as ER visits and Dental Visits.

18

Key Survey Estimates: Heath Care Utilization and health insurance

Quasi-experimental difference-in-differences: comparing in person to mode switch groups on change from fall 2019 to spring 2020

Exiting panel (only 2019 events): Mode switch lower dental events 

Continuing panel (both 2019 and 2020 events): Mode switch lower office provider and physician events, dental care, non-agency home health provider

Health insurance reporting did not differ by group for either panel

Where 2020 events reported in interview for extended vs. continuing panels

Extended panel reported fewer events on the records path and at the event driver

19

We also used our differences in differences analysis to look at changes between the fall 2019 and spring 2020 interviews in the number of events between the in person and mode switch groups. COVID should not have impacted 2019 events. [X]For Panel 23, which only reported on 2019 events, we found the mode switch group reported lower than expected dental events for 2019, but there were no other differences between the groups. [X]For Panel 24, which reported on both 2019 and 2020 events, the mode switch group reported lower office provider and physician events, dental care, and non-agency home health provider visits. It is likely that COVID had an impact on these types of visits. [X]Neither panel showed any difference in the change in health insurance status between the in person and mode switch groups between fall 2019 and spring 2020.

[X]Finally, given the findings above that the extended panel was reporting fewer events than the continuing panel, we wanted to know more about what was happening. There are a number of places in the interview where respondents can report events, to maximize the events that we collect. [X]So we compared the extended panel’s one long Round 6 interview with the ongoing panel’s 2020 events from their Round 3 and Round 4 interviews.

19

Events for in Extended Panel vs. Ongoing Panel
| Events created from...   | Extended(P23 R6) | Ongoing (P24 R3+R4)i | T-test            |
|------------------------|------------------|----------------------|-------------------|
| Records                | 7.289 (0.202)    | 10.539(0.257)        | -9.92(<0.0001) |
| Probes                 | 0.103(0.011)     | 0.143(0.013)         | -2.41(0.0902)     |
| Off-path               | 0.008(0.002)     | 0.010(0.002)         | -1.02(1.0000)     |
| Event follow-up        | 0.051(0.006)     | 0.068(0.008)         | -1.71(0.4732)     |
| Event driver           | 4.257(0.179)     | 6.050(0.223)         | -6.27(<0.0001) |

20

We found that the extended panel reported [X]fewer events from records, keeping with the hypothesis that these respondents may have stopped keeping as complete records as they did before. [X]They also reported fewer events at the event driver, which is the part of the interview where the interviewer reviews events in preparation to collect charge and payment information.

20

Overall Findings

Households switched to telephone less likely to use records than those who did interview in person

No strong differences in response distributions for items using show cards

For health insurance status, no evident impact from mode change or COVID on estimates

Mode switch did not seem to impact event reporting: 2019 did not differ by mode, 2020 tracked with decreases due to COVID

Extended panel reported fewer events than expected

21

So, back to our overall findings. Telephone did seem to have a negative impact on records use, but not on items with show cards. Health insurance didn’t seem to be affected by either mode or COVID, at least in the short term that we looked at in these analyses. Event reporting didn’t seem to be impacted by mode, but we did see COVID-related decreases in some types of health care use. And our concerns about the extended panel’s long reference period were warranted, as they reported fewer events overall and particularly from records for 2020.

21

Future Steps

Re-running 2020 results using full year data

Monitoring records use by mode

Tracking respondent use of show cards

22

I’ll end with a few future steps we’re taking. [X]I mentioned that the 2020 results were based on data through the fall and on data with no editing and makeshift weights. So we are interested in re-assessing our findings with the nicely edited, weighted, and complete full year data.

[X]We are also working on developing monthly reports to track respondent use of records cut by a variety of other variables, so we can address issues while we are in the field. Telephone interviewing is not going away anytime soon, especially since we have now extended two panels for two additional years beyond the normal two-year period, and we are doing almost all of those interviews by phone.

[X]Finally, I mentioned that we don’t have data on when respondents used show cards versus when interviewers have to read them. This coming spring we will start collecting information from interviewers about their observations on respondent use of show cards.

22

Thank You

Alisha Creel

AlishaCreel@westat.com

Photos are for illustrative purposes only. All persons depicted, unless otherwise stated, are models.

23

PHOTOGRAPHY LEGAL DISCLAIMER – DO NOT DELETE THIS SLIDE IF PRESENTATION INCLUDES PEOPLE IMAGES

Thanks so much for watching today, and I hope to see you at the Q&A.

23

Show Card Item Criteria

Asked frequently: maximized the sample size for analysis

Nominal categories: no ordinal scales

Question text does not include show card text: ensured that in an in-person interview, the respondent would only know the show card text by reading it

Variety of placement in the survey: respondents could become fatigued listening to (or reading) show cards as the survey progressed

Long show card text

24

# image2.png

# image3.png

# image4.png

# image19.png

# image13.jpg

# image7.png

# image20.tmp

# image21.tmp

# image22.tmp

# image17.png

# image18.png

# image1.png