Techniques Tested to Improve Representation of Racial and 

Ethnic Minority, Younger, and Maternity Patients in a 

National Hospital Experience of Care Survey

William G. Lehrman, Centers for Medicare and Medicaid Services (CMS)

Marc N. Elliott, RAND

Julie A. Brown, RAND

Katrin Hambarsoomian, RAND

Layla Parast, University of Texas

Megan K. Beckett, RAND

Federal Committee on Statistical Methodology 

2023 Research and Policy Conference

October 24, 2023
Background: Many Patient Groups 

Have Lower Survey Response Rates

• HCAHPS is the first national, standardized, publicly reported survey of patient 

experience with hospital care

– Hospitals choose their survey mode (Mail Only, Phone Only mode, Mail-Phone, Active IVR) 

• Surveys in general, including HCAHPS, often have lower response rates (RRs) for 

adults who are Asian American and Native Hawaiian/Pacific Islander (AA & 

NHPI), Black, Hispanic, and younger

• Methods that improve RRs for these groups are important to ensure that patient 

experiences surveys:

– Fully capture the experiences of all patients

– Adequately measure health equity and equity-targeted quality improvement efforts 

• Because these groups tend to have lower RRs, any effort that improves their RR 

is likely to improve overall representativeness 
We Investigated Two Approaches to Increasing 

Representation of Groups with Lower Response Rates 

• Sequential multi-mode approaches increase RRs and representativeness

– Different patients have different preferred modes of response

– Providing 2 or more modes sequentially allows patients to respond in their preferred mode

– Here we evaluate one 3-mode protocol, three 2-mode protocols, and two single-mode 

protocols

• Longer data collection periods have several potential benefits 

– They facilitate multimode protocols

– They may increase RR

– They may increase representativeness

– Here we test a 49-day data collection period and compare respondents in the 7 days to 

those in the first 42 days
2021 HCAHPS Mode Experiment Design

• 46 participating hospitals

– Sampled 36,001 patient discharges from April 1 to September 30, 2021

– Patients age 18+, overnight stay, surgical/maternity/medical service lines, etc.

– 63% of patients provided email addresses

– Patients randomized within each hospital to 1 of 6 modes

– As in previous HCAHPS mode experiments, survey administration was in English

• Used a 49-day, rather than the usual 42-day field period

• Randomized experiments help compare the representativeness of survey modes

– HCAHPS collects self-reported race/ethnicity, but only from respondents 

– A randomized experiment can show relative differences in RRs by race/ethnicity in de-identified 

data such as HCAHPS 
Mode Day Mail Only Phone Only Mixed Mode Web-Mail Web-Phone Web-Mail-Phone

1 Mail 1st survey Begin phone 

calls Mail survey Email 1st 

invitation Email 1st invitation Email 1st invitation

3 Email 2nd

invitation

4 Email 2nd invitation Email 2nd invitation

6 Email 3rd

invitation Mail survey

7 Email 3rd invitation

8 Mail 1st survey

10 Begin phone calls

21 Mail 2nd survey

28 Begin phone 

calls Begin phone calls

30 Mail 2nd survey

49 End data 

collection

End data 

collection

End data 

collection

End data 

collection End data collection End data collection

Mode Experiment Schedule of Contacts
2021 HCAHPS Mode Experiment Response Rates

Survey Administration Protocol Response Rate

Current HCAHPS Modes

Mail Only 22%

Phone Only 23%

Mail-Phone 31%

Web-first Modes

Web-Mail 29%

Web-Phone 30%

Web-Mail-Phone 36%

• Adding web increased RRs

• Single mode-protocols had the lowest RRs
2021 HCAHPS Mode Experiment Response Rates

by Email Availability

Survey Administration Protocol RR, no available email 

address 

RR, email address 

available

HCAHPS Legacy Modes

Mail Only 21% 24%

Phone Only 20% 23%

Mail-Phone 26% 34%

Web-first Modes

Web-Mail 20% 34%

Web-Phone 19% 37%

Web-Mail-Phone 29% 40%

• Email availability is associated with higher RRs, even for non-web survey modes

• Email availability increases RRs especially for web-first survey modes
Multi-mode Protocols Improve RR and 

Representativeness: Race and Ethnicity

• Web-Mail-Phone had the highest yield for 3 of 5 racial and ethnic groups (and 

the second highest for another) because of its high representativeness and 

overall RR

• Otherwise, the highest or second-highest yield was almost always a 2-mode 

protocol

• Mail Only was the lowest-yield mode for Black, Hispanic, and Multiracial 

patients; Phone Only was lowest-yield for White patients; these modes tie as 

lowest-yield for AA&NHPI patients

• The gains from multi-mode approaches are often 2-3x as large for AA&NHPI, 

Black, Hispanic, and Multiracial patients as for White patients
Multi-mode Protocols Improve RR and 

Representativeness: Age

• Web-Mail-Phone had the highest yield for 6 of 8 age groups and the second 

highest yield for the other two age groups

• Otherwise, the highest or second-highest yield was almost always a 2-mode 

protocol

– Web-Phone was especially successful for ages 18-64

– Web-Mail was especially successful for ages 65-84

• Mail Only had the lowest yield for ages 18-54

• Phone Only had the lowest yield for ages 55+
Multi-mode Protocols Improve RR and 

Representativeness: Service Line and Sex

• Web-Mail-Phone had the highest yield for 4 of 5 combinations of service line 

and sex and the second highest yield for the other group

• Otherwise, the highest or second-highest yield was always a 2-mode protocol

– Web-Mail did especially well for surgical patients

– Web-Phone did especially well for maternity patients

– Mail-Phone did especially well for medical patients

• Mail Only had the lowest yield for maternity patients

• Phone Only had the lowest yield for medical and surgical patients
Best and Worst RRs by Patient Characteristics

Characteristic Lowest RR/Yield Highest RR /Yield

Age

18 - 24 Mail Only Web-Phone 

25 - 54 Mail Only Web-Mail-Phone 

55 - 84 Phone Only Web-Mail-Phone 

85+ Phone Only Web-Mail 

Race and Ethnicity

AA&NHPI Mail Only & Phone Only Mail-Phone & Web-Mail

Black, Hispanic Mail Only Web-Mail-Phone

White Phone Only Web-Mail-Phone

Service Line x Sex

Maternity Mail Only Web-Mail-Phone 

Medical, Surgical (Both Female & Male) Phone Only Web-Mail-Phone 
Extending HCAHPS Data Collection Period from 42 to 49 

Days Improves Representation of Underrepresented Groups

• HCAHPS currently allows patients 42 days after first contact to respond to 

survey

• We tested a 49-day data collection period and compared the final week (days 

43-49) to the first 42 days

– All modes showed meaningful gains in RR in the last week (average +3 pp)

– Biggest gains for underrepresented groups 

• Racial and ethnic minority respondents were 51% of last-week respondents vs. 40% of earlier 

respondents

• Those preferring a language other than English were 13% of last-week respondents vs. 10% of 

earlier respondents.

• Later responses to patient experience surveys are also known to capture poorer care 

experiences than earlier responses
Conclusions

• For HCAHPS, the gains from multi-mode approaches were often 2-3x as large for AA&NHPI, 

Black, Hispanic, and Multiracial people as for White patients 

• Mail Only mode had the lowest yield for Black, Hispanic, Multiracial, age 18-54, and maternity 

patients

• Phone Only mode had the lowest yield for White, age 55+, medical, and surgical patients

• Web-Mail-Phone had the highest yield for most patient groups

• Among two-mode protocols:

– Web-Phone was especially successful for maternity and age 18-64 patients

– Web-Mail was especially successful for surgical and age 65-84 patients

– Mail-Phone was especially successful for medical patients

• While multi-mode approaches consistently outperform single mode approaches, the most 

effective survey modes for a given hospital will depend upon its patient population
May 2023

Questions or comments?

William Lehrman

Government Task Leader, HCAHPS Survey

Centers for Medicare & Medicaid Services (CMS) 

William.lehrman@cms.hhs.gov