Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

HMO CUSTOMER SATISFACTION 

SURVEYS 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 

amended, is to protect the integrity of the Department of Health and Human Services’ (HHS) 

programs as well as the health and welfare of beneficiaries served by those programs. This 

statutory mission is carried out through a nationwide network of audits, investigations, and 

inspectionsconductedby three OIG operating components: the Office of Audit Services, the 

Office of Investigations, and the Office of Evaluation and Inspections. The OIG also informs the 

Secretary of HHS of program and managementproblems and recommends courses to correct them. 

OFFICE OF AUDIT SERVICES 

The OIG’SOffice of Audit Services(OAS)provides all auditing services for HHS, either by 

conducting audits with its own audit resources or by overseeing audit work done by others. Audits 

examine the performance of HHS programs and/or its grantees and contractors in carrying out 

their respective responsibilitiesand are intended to provide independentassessmentsof HHS 

programs and operations in order to reduce waste, abuse, and mismanagementand to promote 

economy and efficiency throughout the Department. 

OFFICE OF INVESTIGATIONS 

The OIG’SOffice of Investigations(01) conducts criminal, civil, and adrnhistrative investigations 

of allegations of wrongdoing in HHS programs or to HHS beneficiaries and of unjust enrichment 

by providers. The investigative efforts of 01 lead to criminal convictions, adrnhistrative 

sanctions, or civil money penalties. The 01 also oversees State Medicaid fraud control units which 

investigate and prosecute fraud and patient abuse in the Medicaid program. 

OFFICE OF EVALUATION AND INSPECTIONS 

The OIG’SOffice of Evaluation and Inspections (OEI) conducts short-term management and 

program evaluations(called inspections)that focus on issues of concern to the Department, the 

Congress, and the public. The findings and recommendationscontained in these inspection reports 

generate rapid, accurate, and up-to-date information on the efficiency, vulnerability, and 

effectivenessof departmental programs. This report was prepared in the New York regional office 

under the direction of Alan S. Meyer, Ph.D., Regional Inspector General. Project staff included: 

REGION II HEADQUARTERS 

Demetra Arapakos, Project Leader Jennifer Antico 

Nancy Harrison Hugh Hetzer 

Brian Ritchie 

Barbara Tedesco 

For additional copies of this report, please contact the New York regional office at (212) 264-

1998. 
Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

HMO CUSTOMER SATISFACTION


SURVEYS


# sERVICq 

J+ “G. 

*+ ‘7 

JUN13 GIBBS BROWN g 

2

u

% 

Inspector General 

% <

++, MARCH 1995 ‘*daa 2 OEI-02-94-O0360 
EXECUTIVE SUMMARY


To assess how Medicare health maintenance organizations (HMOS) are conducting 

customer satisfaction surveys and how they are utilizing the results of these surveys. 

BACKGROUND 

In various staff meetings, the Office of Managed Care in the Health Care Financing 

Administration (HCFA) asked the Office of Inspector General (OIG) to survey how 

Medicare HMOS are measuring customer satisfaction, particularly of Medicare 

beneficiaries, and using the resulting data. The HCFA requested this study in order to 

better ascertain how active its role should be in surveying Medicare HMO enrollees and 

how the surveys HMOS are conducting can be of use to HCFA in its monitoring efforts. 

We selected a stratified random sample of 95 HMO risk and cost contracts out of the 

universe of 185 such contracts with Medicare beneficiaries enrolled as of February 1, 

1995. We sent them a mail questionnaire regarding their customer satisfaction survey 

procedures and their use of survey results. We also requested copies of their survey 

instruments, which we analyzed for content and format. We received 72 completed 

questionnaires and 63 survey instruments. 

FINDINGS 

Virtually All Risk and Cost HMOS Conduct Customer Satisfaction Surveys 

All but one of our respondent HMOS (99 percent) conduct general customer satisfaction 

surveys. 

However, Most HMOS Do Not Target Their Medicare Members 

More than half of the HMOS (55 percent) have never conducted a customer satisfaction 

survey of their Medicare members only. Furthermore, almost all of these (97 percent) 

also do not include questions specific to Medicare members on their general surveys. 

More than one-third of all HMOS (39 percent) do not know the satisfaction rate of their 

Medicare members for their last general survey, and most (65 percent) do not know the 

Medicare response rate. However, the Medicare specific data which is available shows 

that Medicare members have high satisfaction rates. 

HMO Customer Satisfaction Survey Instruments and Procedures Lack Uniformity 

The satisfaction survey instruments used by HMOS vary widely in their format and 

content; in fact, no two are the same. These instruments differ in their 

comprehensiveness and in the rating scales and satisfaction questions used. The survey 

i 
procedures HMOsuse also vary. This lack ofuniformity in HMO surveys renders 

comparisons between HMOS difficult, if not impossible, when assessing Medicare 

beneficiaries’ satisfaction with their plans. 

While in Many Ways Basically Sound, Technical Weaknesses in Many HMO Surveys 

May Mask Problems and Inflate Satisfaction With Managed Care Plans 

Many HMOS appear to be following sound survey principles regarding sampling and 

increasing response rates. Most are also using survey instruments containing clearly 

worded and focused questions (98 percent) and covering a broad range of satisfaction 

dimensions (71 percent.) However, many HMO survey instruments do contain 

weaknesses which may bias, to some degree, survey results. Most significantly, more 

than half (58 percent) include no questions about problems with or complaints about health 

plan services, and twenty-nine pe’rcent use survey instruments which include an 

unbalanced five-point rating scale of three positive, one neutral and only one negative 

response category. Most HMOS are also lacking mail follow-up procedures (21 of 32 

which use mail surveys) and are not conducting non-respondent analyses (87 percent.) 

One-third of HMOS scored adequate on an index of instrument adequacy, and less than 

half (44 percent) scored adequate on an index of procedure adequacy. 

HMOS Use Their Survey Results as Much for Marketing as for Quality Improvement 

Nearly three-fourths of HMOS (74 percent) say they use the results of their customer 

satisfaction surveys to market themselves to potential new members. A similar number 

(76 percent) report also using their customer satisfaction data to develop quality 

improvement or corrective action plans. A majority of HMOS share their survey findings 

with physicians (76 percent) and with plan members (67 percent.) 

IMPLICATIONS FOR MEDICARE 

We believe the usefulness of customer satisfaction surveys as currently conducted by 

HMOS is substantially reduced by their lack of uniformity, limited focus on Medicare 

beneficiaries and technical weaknesses. Therefore, if HCFA seeks data on the satisfaction 

of Medicare beneficiaries with managed care, we believe it can not rely upon industry 

surveys as they are now conducted. The HCFA may want to consider alternative 

approaches to measuring Medicare client satisfaction with managed care, such as 

conducting its own surveys or requiring HMOS to periodically survey their Medicare 

members with a standardized instrument and comparable procedures. 

The Office of Inspector General is planning further work surveying Medicare beneficiaries 

about their experiences with HMOS. We will once again conduct a survey of Medicare 

HMO members similar to one already completed. We are also working on a technical 

assistance report which will provide HCFA with a more detailed discussion of our 

methodology for conducting beneficiary surveys and will identify useful survey techniques 

and methods based on our prior experience. 

ii 
COMMENTS 

Wereceived favorable comment.s from HCFAontie drafi repoti, expressing their 

appreciation for the timeliness and significance of our study, In particular, HCFA states 

that this report will be a major factor in influencing its decision to develop its own 

beneficiary satisfaction survey capability. The actual comments received are included in 

Appendix C. 

... m 
TABLE OF CONTENTS 

PAGE 

EXECUTIVE SUMMARY 

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1


FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...5


� Virtually All HMOs Conduct General Surveys . . . . . . . . . . . . . . . . . . . . . ..5


� Most HMOs Do NotTargetMedicare Members . . . . . . . . . . . . . . . . . . . ...5


� HMO Surveys Lack Uniformity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...6


� Technical Weaknesses May Mask Problems and Inflate Satisfaction . . . . . . . . . . .7


� HMOs Use Survey Results as Much for Marketing as for Quality Improvement. . . .9


IMPLICATIONS FOR MEDICARE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10


APPENDICES 

A: Statistical Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. A-1


B: Indexes of Survey Adequacy .,. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. B-l

INTRODUCTION 

PURPOSE 

To assess how Medicare health maintenance organizations (HMOS) are conducting 

customer satisfaction surveys and how they are utilizing the results of these surveys. 

BACKGROUND 

In various staff meetings, the Office of Managed Care in the Health Care Financing 

Administration (HCFA) asked the Office of Inspector General (OIG) to survey how 

Medicare HMOS are measuring customer satisfaction and using the resulting data. The 

OIG was asked to pay particular attention to what the managed care industry is doing to 

measure the satisfaction of its Medicare members. The HCFA requested this study in 

order to better ascertain how active its role should be in surveying Medicare HMO 

enrollees and how the surveys HMOS are conducting can be of use to HCFA in its 

monitoring efforts. 

This report follows earlier OIG reports on Medicare managed care. These included one 

entitled “Beneficiary Perspectives of Medicare Risk HMOS” which surveyed beneficiaries 

about their experiences with HMOS. This report found that while risk HMOS provide 

adequate service access for most of their Medicare enrollees, some serious problems 

remained with enrollment procedures and service access and disenrollees were more likely 

than enrollees to have experienced problems with HMO services. A further report on 

“Medicare Risk HMOS: Beneficiary Enrollment and Service Access Problems” looked 

more closely at HMO-level data to identify the distribution of these problems and found 

that they existed in varying degrees of intensity among HMOS and more frequently with 

disenrollees than with enrollees. Another report, entitled “Medicare Risk HMO 

Performance Indicators, ” found that HMO disenrollment rates, along with customer 

satisfaction surveys, appear to be useful managed care performance indicators. 

l%e Industry 

Health maintenance organizations are a form of managed care in which a patient selects a 

primary care physician from a group of approved plan providers to act as the patient’s 

first point of contact within the health care system. This physician must authorize any 

specialist, hospital or other type of care the patient receives. According to industry 

estimates, approximately 50 million individuals in the United States are enrolled in one of 

the 550 HMOS across the country, 

An HMO can pay its physicians in different ways. A primary care physician or specialist 

can be paid on a cavitation basis, in which he or she is paid one monthly amount per each 

patient regardless of how much care that patient receives. Physicians can also be paid on 

a fee-for-service basis. Some HMOS use a combination of these different payment 

methods. 

1 
Medicare HMOS 

Medicare beneficiaries have the option of receiving their health care from an HMO 

approved by HCFA. Once approved, the HMO generally applies for a risk or cost 

Medicare contract. In a risk contract, the HMO provides the full Medicare benefit 

package andispaid onaprospective percapita basis, inwhich itisrequired to absorb any 

financial losses butispemitied toremin any fimncial savings. Under such a contract, 

payment ismadeon aprepaid cavitation basis with no retroactive adjustment. The HCFA 

encourages HMOS to apply for risk contracts. In a cost contract, the HMO also provides 

the full Medicare benefit package but is paid on a reasonable cost basis. An HMO can 

also serve Medicare patients through a health care prepayment plan agreement or as a 

demonstration project. 

The number of Medicare risk and cost contracts continues to grow. As of February 1, 

1995, 157 plans had risk contracts and 30 had cost contracts with HCFA, with a total 

enrollment of approximately 2.5 Medicare million beneficiaries. This is an increase from 

July of last year, when there were 136 risk HMOS and 27 cost HMOS. In February 1993, 

only 87 HMOS had risk contracts to serve Medicare beneficiaries. 

While only about seven percent of the Medicare population nationwide is enrolled in 

HMOS, the geographical distribution of tlis enrollment varies widely. The distribution of 

Medicare HMO enrollees is concentrated in four States: California, New York, Florida, 

and Arizona. A few States have no Medicare beneficiaries enrolled in managed care 

programs. 

Section 42 CFR 417. 107(h) of the regulations require federally qualified HMOS with 

Medicare enrollees to implement ongoing quality assurance programs. These programs 

must have the following basic components: a quality assurance methodology, a peer 

review process, systematic data collection of performance and patient results, and 

remedial action procedures. 

As part of the systematic data collection requirement listed above, HMOS are required to 

share their data collection results with their providers of care and institute any needed 

changes based on these results. Data can be collected from any of several different 

sources, including member satisfaction surveys. These surveys, however, are not required 

by law. 

Customer Satisfaction Initiatives 

Several efforts have been under way in the managed care industry to develop 

methodologies and instruments for measuring and reporting performance ratings, including 

customer satisfaction, in managed health care. These initiatives have primarily been 

motivated by an interest in the industry and among health care consumers in establishing 

standard measures by both individuals and employers can compare and contrast different 

health plans. Few of these initiatives, however, are specifically aimed at the Medicare 

population. 

2 
Two industry groups are particularly active in managed care quality and customer 

satisfaction issues. The National Committee for Quality Assurance (NCQA) is a voluntary 

private accreditation agency active in setting and enforcing HMO quality standards. In 

November 1993, it developed the Health Plan Employer Data and Information Set 

(HEDIS), which defines performance measures to evaluate, among other things, a plan’s 

quality of care, member access to care, and member satisfaction. The NCQA publishes 

the results of its accreditation reviews nationwide. Also, the Group Health Association of 

America (GHAA), a managed care industry group to which most HMOS belong, has 

developed a model consumer satisfaction questionnaire which is available to its members. 

METHODOLOGY 

In conducting this inspection, we selected a stratified random sample out of the universe 

of 185 HMO risk and cost contracts with Medicare beneficiaries enrolled as of February 

1, 1995 (nine additional HMO risk and cost contracts were dropped from the universe 

because they had no Medicare enrollees at the time the sample was drawn.) These HMO 

contracts were stratified into three groups of high, medium and low Medicare enrollment, 

based on the number of Medicare beneficiaries in each. The HMOS in the high stratum 

have Medicare enrollments of larger than 42,550, while those in the medium and low 

strata have between 42,500 and 1000 and less than 1000 Medicare members respectively, 

We purposely contacted all of the contracts in the high stratum. See Appendix A for a 

more detailed explanation of strata selection. 

We selected 95 HMO risk and cost contracts for the fiml sample: all 13 from the first 

stratum, 70 from the second stratum, and 12 from the third stratum. Medicare enrollment 

in the 13 high stratum contracts accounts for 51 percent of all Medicare enrollment 

nationwide in managed care. Forty-seven different HMOS across the country hold these 

95 contracts, Eighty-two of the 95 are risk contracts, and the remaining 13 are cost 

contracts. 

We sent all 95 a mail questionnaire which requested information about their customer 

satisfaction survey procedures and their use of survey results. We also requested copies 

of the HMOS’ survey instruments. After allowing six weeks for data collection, during 

which time we conducted a second mailing to non-respondents, we achieved an overall 

response rate of 76 percent for the questionnaires. We also achieved a 66 percent overall 

response rate for the survey instruments after making a minimum of two follow-up 

telephone calls to HMOS who did not initially send us their instruments. For the mail 

questionnaires, we achieved response rates of 100 percent for the high stratum, 77 percent 

for the medium stratum, and 42 percent for the low stratum. For the survey instruments, 

we achieved response rates of 100, 64 and 42 percent for each of the three strata 

respectively. All differences reported between strata are statistically significant at the 95 

percent conildence level. 

Differing response rates among strata suggest the possibility of non-response bias. While 

we did use Chi-square to test for such bias, due to our relatively small sample size and 

3

resulting small cell sizes in the two-variable tables, it was not a valid test. Therefore, we 

acknowledge the possibility of non-response bias but are not able to quanti@ it. 

To assess HMO customer satisfaction survey procedures, we reviewed the returned mail 

questionnaires to determine, among other things, their sampling methods, follow-up 

procedures, and their means of determining response and satisfaction rates. Our findings 

on survey procedures are based on the HMOS’ answers to these questionnaires. See 

Appendix A for contldence intervals of fifteen key questions. 

To assess HMO survey instruments, we developed a detailed review sheet which we used 

to systematically evaluate each instrument for both form and content. This review sheet 

included assessments of each instrument’s length, dimensions of satisfaction measured, 

scales and format used, clarity of instructions and questions, and user friendliness. 

We also constructed two indexes of survey adequacy - one for instruments and the other 

for procedures. We used data from the instrument review sheets discussed above to 

construct the index of instrument adequacy. This index was based on three key variables: 

comprehensiveness of satisfaction dimensions, balanced response categories, and problem 

specific questions. The index of procedure adequacy was based on four variables: level of 

confidence sought, frequency of follow-up efforts, conducting a non-respondent analysis, 

and use of balanced criteria for determining overall satisfaction. For these four variables, 

we used indicators from the questionnaires returned by HMOS. In each of the two 

indexes, we gave the variables a subscore, which were then combined to give total scores 

for instrument adequacy and procedure adequacy respectively. The indexes of survey 

adequacy are explained in greater detail in Appendix B. 

This inspection was conducted in accordance with the Quality Standards for Inspections 

issued by the President’s Council on Integrity and Efficiency. 

4

FINDINGS 

VIRTUALLY ALL RISK AND COST HMOS CONDUCT CUSTOMER 

SATISFACTION SURVEYS 

General Membership S@”sfaction Surveys 

All but one of the respondent HMOS (99 percent) conduct general customer satisfaction 

surveys. Almost all (95 percent) consider these general surveys to be very or somewhat 

useful. A majority of HMOS (60 percent) conduct their surveys at least once a year, 

while thirteen percent survey their members twice a year. Most of the remaining 28 

percent conduct satisfaction surveys on an ongoing basis. 

Nearly three-fourths of HMOS occasionally (24 percent) or always (49 percent) use a 

professional research agency to conduct their customer satisfaction surveys, with a total of 

37 different research fms cited by the responding HMOS. All of the HMOS with high 

Medicare enrollment hire professional research firms, as compared to 70 percent of 

HMOS with medium and low Medicare enrollments. 

Other ~pes of Surveys 

A majority of HMOS are also conducting other types of surveys. These include surveys 

of disenrollees (92 percent), on the functional or health status of their members (58 

percent), and of the working aged (53 percent.) Fifty-nine percent of HMOS also survey 

the physicians who work for them about their satisfaction. 

HOWEVER, MOST WOS DO NOT TARGET THEIR MEDICARE MEMBERS 

Medicare Sti”sfaction Surveys 

While HMOS conduct satisfaction surveys of their general memberships, they pay less 

specific attention to their Medicare members. More than half of the HMOS (54 percent) 

have never conducted a customer satisfaction survey of their Medicare members only; in 

fact, eight percent of these survey only their commercial, non-Medicare, members on 

satisfaction. Seven of the HMOS who have not yet conducted a Medicare only survey did 

volunteer, however, that they are planning to do so within the next year. Almost all of 

the HMOS (97 percent) which do not conduct Medicare only surveys also do not include 

questions specific to Medicare members on their general surveys; the few who do ask only 

a limited number of Medicare questions. 

While most HMOS have the potential to identify Medicare members in their general 

surveys by identifying the respondent’s age, for the most part they are not extracting 

Medicare specific data. More than one-third of all HMOS (39 percent) do not know the 

satisfaction rate of their Medicare members for their last general survey, and most (65 

percent) do not know the Medicare response rate. Nevertheless, whether or not they 

5 
survey Medicare enrollees, more than half of the HMOS (58 percent) believe it is easier to 

survey Medicare members than it is to survey non-Medicare members, primarily because 

the former are more responsive, have more time, and are easier to reach. 

The Medicare data which is available shows that Medicare HMO members have high 

satisfaction rates. All of the HMOS with Medicare data report overall satisfaction rates of 

75 percent or higher for their Medicare members. Similarly, virtually all (99 percent) 

report the same 75 percent or higher satisfaction rate for all members. 

HMO CUSTOMER SATISFACTION SURVEY INSTRUMENTS AND 

PROCEDURES LACK UNIFORMITY 

The lack of uniformity in HMO survey instruments and procedures renders comparisons 

between HMOS difficult, if not impossible, when assessing Medicare beneficiaries’ 

satisfaction with their plans. 

Satisfaction Survey Instruments 

The survey instruments used by HMOS vary widely in their format and content. No two 

risk and cost HMOS use the same instrument. Of our sample survey instruments, none 

use an exact duplication of the GHAA satisfaction survey instrument, although almost half 

(48 percent) use some exact or similar headings, questions and rating scales from that 

survey. The HMO instruments differ significantly in their length, ranging in size from 1 

to 45 pages and including from between 9 to 159 different questions. 

Rating scales and overall satisfaction questions on survey instruments are also inconsistent 

across HMOS. Our review of HMOS’ instruments identified 26 different scales used for 

responses, ranging from a simple two-point scale of yes or no, to a ten-point scale 

covering a range of satisfaction levels. Fourteen different types of questions are used to 

measure overall satisfaction, including questions which ask about satisfaction with medical 

care, most recent visit, quality of service, health plan and a particular medical center. 

Three-fourths of HMOS, however, use a question about satisfaction with their plan to 

measure overall satisfaction. 

Sti”sfaction Survey Procedures 

Sampling procedures also differ. Fourteen percent of HMOS survey their entire 

membership. Of the 86 percent who survey a sample of their membership, sample sizes 

vary from 100 to over 5000. The size and type of sample used varies according to the 

purpose of each HMO’s survey. While most (51 percent) use a simple random sample, 

another 41 percent use a stratified random sample, and eight percent a purposive sample. 

Forty-nine percent of the HMOS select their sample from a universe of one particular 

subgroup, as defined, for example, by a minimum length of membership, while most of 

the remaining sample from the universe of their entire membership. 

6

Furthermore, other survey procedures used by HMOS vary. Forty-four percent administer 

their customer satisfaction survey by telephone only, 16 percent administer it by telephone 

and mail, and 37 percent by mail only. The remaining generally use an in-office selfadministered questionnaire in combination with mail or telephone. Excluding those which 

conduct ongoing surveys, sixty-two percent of HMOS stop data collection after ten weeks 

and the remaining third continue collecting data for longer than ten weeks. 

WHILE IN MANY WAYS BASICALLY SOUND, TECHNICAL WEAKNESSES IN 

MANY HMO SURVEYS MAY MASK PROBLEMS AND INFLATE 

SATISFACTION WITH MANAGED CARE PLANS 

HMO Survey Strengths 

Many HMOS appear to be following sound survey principles regarding sampling, efforts 

to increase response rates by telephone, and instrumentation. A majority (71 percent) 

were seeking conildence intervals of 95 percent or higher in designing their sample size. 

Close to half (44 percent) used sample sizes of over 1000 members, which, in 

combination with their generally high response rates, should have ensured a good level of 

precision in their surveys. Additionally, most HMOS who use telephone surveys make a 

strong effort to maximize their response rates. Almost all of these (94 percent) try to 

contact members at least three times before considering them to be non-respondents. 

Furthermore, our review of HMO survey instruments revealed several positive features. 

A great majority of the instruments generally include questions which are specific (92 

percent), clearly worded (98 percent) and focused on only one thought at a time (98 

percent.) The instruments also cover a broad range of satisfaction dimensions. The most 

common include ability to provide prompt service (96 percent), overall satisfaction with 

services (87 percent), staff courtesy (84 percent), access to services (82 percent), 

physician communication (81 percent), and physician competence (77 percent.) Of the 

seventeen possible dimensions of satisfaction we looked for in the HMOS’ instruments, a 

majority (71 percent) include at least ten. 

HMO Survey Instrument Weaknesses 

However, our instrument review also revealed some weaknesses in the survey instruments 

which may bias, to some degree, survey results. More than half of the instruments (58 

percent) include no questions about problems with or complaints about health plan 

services. Of the 42 percent which do include these topics, most ask only one or two 

questions. For the most part, the questionnaires ask only if a member has ever had a 

problem with or complaint about their care, and how satisfied he or she was with its 

resolution. The OIG report referenced earlier on beneficiary perspectives of risk HMOS 

included several questions in its beneficiary survey specific to problems with HMO 

services. These questions were important in understanding, among other things, reasons 

for dissatisfaction and disenrollment with HMOS. Half of the instruments (51 percent) do 

not ask respondents for their suggestions for improving the HMO. 

7

Other deficiencies noted in the HMOS’ instruments may compromise their objectivity. 

Forty-six percent use at least one unbalanced rating scale. The most common of these, a 

five-point rating scale of three positive, one neutral and only one negative response 

category, is used by twenty-nine percent of the HMOS. Furthermore, of the quarter 

which use agree or disagree statements, almost all include only positive statements for 

respondents to respond to. While this has the advantage of ensuring greater ease for the 

respondent, it also has the disadvantage of possibly resulting in more positive ratings. 

Other weaknesses noted include few or unclear interviewer instructions on telephone 

survey instruments (24 percent), confusing or unclear questionnaire formats (20 percent), 

and inconsistent rating scales (20 percent) and repetitive questions within an instrument 

(15 percent.) 

We also rated each survey instrument to determine its level of user friendliness. In doing 

this, we looked for several qualities, including clarity of format, simplicity of directions 

and questions, and overall attractiveness. While most surveys (73 percent) were rated 

user friendly, only eleven percent were rated very user friendly and 16 percent were 

deemed not user friendly. A majority (80 percent) of those rated user friendly are mail 

surveys, while most (70 percent) rated not user friendly for either the respondent or the 

interviewer are telephone surveys. 

HMO Survey Procedure Weaknesses 

Most HMOS are also lacking certain survey procedures which, if used, may increase 

survey accuracy. Twenty-nine of 32 sample HMOS which use mail surveys do not try to 

contact members who do not return their original questionnaires; just one-third send nonrespondents a second copy. Additionally, a large majority (87 percent) have never 

conducted a non-respondent analysis, which would enable them to know if nonrespondents differed in any way from respondents, thus alerting them to possible bias in 

the survey findings. Finally, less than half of the HMOS (43 percent) have ever 

conducted a bilingual survey. All of these HMC)S have conducted surveys in Spanish, and 

many are based in States with a large Spanish-speaking population. The lack of a 

bilingual survey may be a survey weakness in those geographical areas with large numbers 

of non-English speaking clients; without them, the experiences and satisfaction of these 

clients can not be fully comprehended. 

Indexes of Survey Adequacy 

As described in our methodology and explained in Appendix B, we constructed two 

indexes of survey adequacy - one for instruments and the other for procedures. One-third 

of the HMOS’ instruments (32 percent) scored adequate on our survey instrument index, 

while another 28 percent scored somewhat adequate. Of the remaining instruments, 24 

percent scored somewhat inadequate and 16 percent inadequate. 

The HMOS were also scored on the adequacy of their survey procedures. On this index, 

less than one-half (44 percent) scored adequate, and thirty-one percent scored somewhat 

8


—-
adequate. Another ten percent of the HMOS scored somewhat inadequate, while 15 

percent scored inadequate. 

HMOS USE THEIR SURVEY RESULTS AS MUCH FOR MARKETING AS FOR 

QUALITY IMPROVEMENT 

Marketing 

A majority of HMOS are using their satisfaction surveys in their marketing plans. Nearly 

three-fourths (74 percent) say they use the results of their customer satisfaction surveys to 

attract potential new members. Most of this marketing consists of reporting survey data 

in speeches or other oral presentations, and in pamphlets or brochures. Seventeen percent 

use their survey findings for advertising purposes. The HMOS which use their survey 

results for marketing have higher overall satisfaction rates than those which do not market 

their survey results; 46 percent of the former report overall satisfaction rates of 75 percent 

or more, while 20 percent of the latter report the same overall satisfaction rates. 

Quality Improvement 

A similar number of HMOS utilize survey results for quality improvement purposes. 

Three-fourths (76 percent) report using their customer satisfaction data to develop 

improvement or corrective action plans. One-third (34 percent) use the results for 

tracking performance and developing strategic goals. Almost all (97 percent) include their 

satisfaction surveys in their Medicare quality assurance programs. 

Most HMOS also share their survey results with employees and members. A majority 

distribute their survey findings to physicians (76 percent) and to plan customers (67 

percent.) In fact, more than one-third (36 percent) use these results to determine all or 

part of their physician reimbursement. Of those HMOS who conduct satisfaction surveys 

of their physicians, a few (16 percent) compare these results to those of their member 

satisfaction survey results. 

Differences Between HMOS 

The HMOS use their survey results somewhat differently depending on the size of their 

Medicare enrollment. Almost all of the HMOS with high Medicare enrollment (92 

percent) use their survey data for quality improvement purposes, compared to 75 percent 

of HMOS with medium and low Medicare enrollments. Marketing of survey results is 

conducted by 58 percent of high enrollment HMOS, in contrast to 75 percent of medium 

and low enrollment HMOS. A comparison of the instruments used by high Medicare 

enrollment HMOS with those used by medium and low Medicare enrollment HMOS 

reveals some important differences in this regard. For example, more than half of the 

former (69 percent) have questions which ask members about problems or complaints with 

their health plans, as compared to just forty percent of the latter, Also, HMOS with a 

large number of Medicare enrollees are more likely to ask for suggestions for 

improvement on their questionnaires than are HMOS with fewer Medicare enrollees. 

9 
We believe the usefulness of customer satisfaction surveys as currently conducted by 

HMOS is substantially reduced by their lack of uniformity, limited focus on Medicare 

beneficiaries and technical weaknesses. Therefore, if HCFA seeks data on the satisfaction 

of Medicare beneficiaries with managed care, we believe it can not rely upon industry 

surveys as they are now conducted. The HCFA may want to consider alternative 

approaches to measuring Medicare client satisfaction with managed care, such as 

conducting its own surveys or requiring HMOS to periodically survey their Medicare 

members with a standardized instrument and comparable procedures. 

The Office of Inspector General is planning further work surveying Medicare beneficiaries 

about their experiences with HMOS. We will once again conduct a survey of Medicare 

HMO members similar to one already completed. We are also working on a technical 

assistance report which will provide HCFA with a more detailed discussion of our 

methodology for conducting beneficiary surveys and will identify useful survey techniques 

and methods based on our prior experience. 

COMMENTS 

We received favorable comments from HCFA on the draft report, expressing their 

appreciation for the timeliness and significance of our study. In particular, HCFA states 

that this report will be a major factor in influencing its decision to develop its own 

beneficiary satisfaction survey capability. The actual comments received are included in 

Appendix C. 

10

APPENDIX A 

STATISTICAL PROCEDURES 

z. SAMPLE SELECTION 

Our original universe of HMO Medicare contracts consisted of 193 risk and cost contracts 

which had been awarded as of February 1, 1995. However, we eliminated eight of these 

contracts which had no Medicare enrollees at the time our sample was drawn. This left us 

with a universe of 185 risk and cost contracts. 

We stratified these 185 contracts into three groups - large, medium, and small - based on 

the number of Medicare enrollees in each HMO contract. Just 13 large HMO contracts 

accounted for 51 percent of all Medicare managed care enrollment. We sampled these 13 

at a rate of 100 percent. We sampled the medium stratum HMOS at a rate of 54 percent 

and the low stratum HMOS at a rate of 28 percent. 

The sample size of 95 was based on the assumption of a 75 percent response rate. The 95 

were stratified as follows: 

STRATA # OF MEDICARE ENROLLEES UNIVERSE SAMPLE 

1 (large) > 42500 13 13 

2 (medium) 1000-42500 130 70 

3 (low) <1000 42 12 

TOTAL 185 95 

1

II. CONFIDENCE INTERVWS FOR 15 KEY QUESTIONS 

We calculated conildence intervals for fifteen key questions. The response estimate and 

95% confidence interval is given for each of the following: 

From the mail cwestionnaire 

1.	 Have you ever conducted a customer satisfaction survey of any kind? 

“Yes” response estimate: 99% 

Lower interval: 97% 

Upper interval: 100% 

2.	 Have you ever conducted a customer satisfaction survey of your Medicare 

enrollees only? 

“No” response estimate: 55% 

Lower interval: 44% 

Upper interval: 66% 

3.	 How many times did you try to contact members who did not return the mail 

questionnaire before finally considering them to be non-respondents? 

“No contact” estimate: 65% 

Lower interval: 49% 

Upper interval: 80% 

4.	 How many times did you try to reach members by telephone before finally 

considering them to be non-respondents? 

3 or more times estimate: 94% 

Lower interval: 89% 

Upper interval: 98% 

5.	 Did your survey instrument include any questions which were asked only of 

Medicare enrollees? 

“No” estimate: 97% 

Lower interval: 93% 

Upper interval: 100% 

6.	 What conildence level were you seeking? 

95% or higher estimate: 71% 

Lower interval: 57% 

A-2

7, 

8. 

9. 

10. 

11. 

12. 

Upper interval: 85%


For your Medicare enrollees only, what response rate did you achieve in your last


customer satisfaction survey?


“Don’t Know” estimate: 65%


Lower interval: 54%


Upper interval: 76%


What percentage of your Medicare members only did you fiid to be satisfied


overall?


“Don’t Know” estimate: 39%


Lower interval: 30%


Upper interval: 49%


Did you use the results of your customer satisfaction survey to develop a quality


improvement or corrective action plan?


“Yes” estimate: 76%


Lower interval: 67%


Upper interval: 86%


Did you share the results of your customer satisfaction survey with your members?


“Yes” estimate: 67%


Lower interval: 57%


Upper interval: 77%


Did you share the results of your customer satisfaction survey with the physicians


who work for your HMO?


“Yes” estimate: 76%


Lower interval: 66%


Upper interval: 87%


Did you use the results of your customer satisfaction survey to market the HMO to


potential new members?


“Yes” estimate: 73%


Lower interval: 65%


Upper interval: 83%


From the survey instrument review sheet 

13.	 Generally, are [instrument] questions worded clearly and focused on one thought? 

“Yes” estimate: 98% 

Lower interval: 96% 

Upper interval: 100% 

14.	 Did the instrument use a rating scale of poor, fair, good, very good, and excellent? 

“Yes” estimate: 29% 

A-3

Lower interval: 19% 

Upper interval: 39% 

15.	 How many questions, including subquestions, ask about problems with or 

complaints about health plan services? 

“No questions” estimate 58% 

Lower interval: 45% 

Upper interval: 71% 

APPENDIX B 

INDEXES OF SURVEY ADEQUACY 

I. INDEX OF INSTRUMENT ADEOUACY 

This index of instrument adequacy is based on the sum of scores achieved by each HMO 

on three separate variables: 

(1) comprehensiveness of satisfaction dimensions 

(2) balance of response categories 

(3) problem-specific questions 

Scores on each variable could range from +2 to -2 according to the criteria for each 

discussed below. 

(1) Comprehensiveness of S~”sfaction ~“mensions 

The following ten dimensions of satisfaction were selected from a longer list of 

such dimensions which we looked for in the HMOS’ survey instruments: patient 

access to general or specialty care; physician or staff courtesy; cost of care; 

coordination or continuity of care; physician competence; choice of providers; 

communication with physicians or staffi office waiting time; suggestions for 

improving HMOS; and, plans to stay with the plan. 

Based on the number of these dimensions included in each HMO’s survey 

instrument, the following scores were used: 

No. of dimensions Score 

o-4 -2 

5 -1 

6-7 +1 

8-10 +2 

(2) Bahznce of Response Categories 

A-1

Balance of response categories refers to whether or not a rating scale includes an 

even number of positive and negative responses. For example, a scale with one 

positive, one neutral, and one negative response - good, fair, poor - is balanced, 

while a scale with two positive, one neutral, and one negative response - very 

good, good, fair, poor - is unbalanced. Scores for this variable were based on two 

items: 1) whether response categories for overall satisfaction questions were 

balanced or not; and 2) whether response categories for all other questions were all 

balanced, almost all balanced, only some balanced, or all unbalanced. Based on 

the extent to which HMOS used balanced categories, the following scores were 

used: 

OVERALL SATISFACTION RESPONSE CATEGORIES 

ALL OTHER RESPONSE


CATEGORIES


All Balanced


Almost All Balanced


Only Some Balanced


None Balanced


All Not All No Overall 

Balanced Balanced Question 

+2 -1 +2 

+1 -1 -1 

-1 -2 -2 

-2 -2 -2 

(3) Problem-Specific Questions 

Based on the number of questions about potential problems or complaints included 

in the HMOS’ survey instruments, the following scores were used: 

# of Ouestions Score 

o -2 

1-4 +1 

5+ +2 

After scoring the HMOS on each of the three variables, we added the three scores together 

to give a total instrument adequacy score for each HMO. Scores ranged from +6 to -6, 

with a midpoint of +1.65. The mean score is 1.1. 

The 63 HMO instruments were grouped into the following four levels of adequacy based 

on their index scores: 

Level of Adeauacv #HMOs A Score 

Adequate 20 32% (+4 to +6) 

Somewhat adequate 13 28% (+1 to +3) 

Somewhat inadequate 20 24% ( o to -2) 

Inadequate 10 32% (-3 to -6) 

II. INDEX OF SURVEY PROCEDURE ADEOUACY 

B-2

This index of procedure adequacy is based on the sum of scores achieved by each HMO 

on four separate variables: 

(1) level of confidence sought


(2) frequency of follow-up efforts


(3) conducting a non-respondent analysis


(4) use of balanced criteria for determining overall satisfaction


Scores on each variable could range from +2 to -1 according to the criteria for each 

discussed below. 

(1) Level of Conj7dence Sought 

Based on the level of confidence HMOS were seeking when selecting their survey 

samples, the following scores were used: 

Confidence Interval Score 

90% or higher +1 

under 90% -1 

Does not apply 

(did not use sample) o 

(2) Frequency of Follow-up Efforts 

The extent to which HMOS employed follow-up procedures with non-respondents 

was scored for each of three survey modes: mail, telephone and in-office. The 

following scores were used: 

A. Mail: 

# times non-rest)ondents are contacted: Score 

2 or more times +2 

1 time +1 

No contact made -1 

Does not apply (no mail survey) o 

B. Telephone: 

# times non-respondents are contacted: Score 

3 or more times +2 

1 or 2 times +1 

No contact made -1 

Does not apply (no phone survey) o 

c. In-office survey: 

B-3

Follow-ut) mocedures Score 

Follow-up with refisals +2 

No follow-up with refhsals -1 

Does not apply (no in-office survey) o 

None of the HMOS used all three modes. However, those which used both mail and 

telephone to administer their survey were given one combined score for follow-up of 

frequency efforts. These HMOS could not achieve a score greater than +2; in this way, 

they were not given extra credit for using two survey modes. If they achieved a negative 

score for one of the two modes, it was subtracted from any positive scores to give a total 

score. 

(3) Conducting a Non-respondent Analysis 

Based on whether an HMO conducted a non-respondent analysis, the following 

scores were used: 

Non-respondent analvsis? Score 

Yes +1 

No -1 

(4) Use of Balanced Criteria for Determining Overall Satl”sfaction 

On the mail questionnaires, we asked HMOS how they derived an overall 

satisfaction rating for their members. If they determined this rating by taking only 

the positive responses from a scale which itself is balanced, they used balanced 

criteria to derive overall satisfaction. Based on whether an HMO used balanced 

criteria in determining an overall satisfaction rate, the following scores were used: 

Balanced Criteria? Score 

Yes +1 

No -1 

As with the index of instrument adequacy, we added an HMO’s scores on these four 

variables together to give a total procedure adequacy score for each HMO. Scores ranged 

from +5 to -5, with a midpoint of 2. The mean score is 1.4. 

The 72 HMO questionnaires on survey procedures were grouped into the following four 

levels of adequacy based on their index scores: 

Level of Adequacy #HMos x Score 

Adequate 32 44% (+5 to +3) 

Somewhat adequate 22 31% (+2 to +1) 

Somewhat inadequate 7 10% ( o to -1) 

Inadequate 11 15% (-2 to -5) 

B-4

The Adt]]tnijtrator 

Washington, D.C. -213201 

. 

DATE: 

To: 

FRONI: 

sw13JEc’~: Ofke Of ~spector Gct\cral Drafi Rcpofl: “Health MaintenaL\ce 

@:tiations @MOs) CUs[oIner Satisfaction Surveys,” 

(OET:-02-94-00350) 

We rw.kved the ~.hove-referenced && r,:po~ Wtich assesse:; how HbIOS are 

COndUCfig Custon:er satisfaction su~eys ;tnd ho~~they are ub~g the reSuhS Of th~.$e 

S~eYS The s~d~ WrISconducted in coctperation with HCFA7S man:~:cd care o~cmtious 

and has provided Inajor insights jnto the need for ~ i.n&pendent Offi,;e”of Ma.nagecl Care 

bcnefi;itiy survey capabi~~, Ourdetauedcolmentsareattzchcdt 

Think you for the opporh~fi$ to revie~v and co[~nel~t on tltis draft report. 

Attachment 

. 

. 

. 

. 
.

— 

. - — 

..” ­

~&Mhumlkk ,. 

“TIIe OIG study has provided major insight:; into the need for ml “independent oK~c~ of 

Managed Care lxn :fici~ sumey capabifil~ ~d on the we of issuesl:;wey questions 

our pro~am should include. 

The study is nletilodolo5cal\y sound and the . sarn~le sizelrespot~se , tnd suppofled rate large by, enou@ tie studyo

assure statistical si;piflctice. The conclu;tons are based on 

data. Find\ngs of pdcdu iniierest to us include: 

1;	 Wile most HMOS conduct member satisfaction SWV% most@ ‘t’c’ude 

Mcdicml: members in the surve)s. ,’ 

2. Hh10 survey instnwlents lack Utifonfity of approach and dl:si~~. 

3, Tecb~id wea.hess’:s give ~ o~ef.~ated SCnSC of mernbec satisfaction and 

may ma!;k PrOb@sI ~ 

4. Survey c.ata are used as much fcr marketing pUJ_POS~SaS for q~laliy 

imrmove:ment purpo:;es. . 

Ttti~ study will be a major factor in influellckg our decision to clevelo? WC would like to express ow appreciation an “mdependel~tto 

beneficiary satisfx:tiou sines capabui~. 

the OIG for ad&e\ sing our operational nctds in such a timely mumer. \Ye hop~ fi~ 

futu.Tt will intl~~dc stilar joint inte~agenty cooperation. 

..