Department  of  Health  and  Human  Services 
OFFICE  OF 
INSPECTOR  GENERAL 
HMO  CUSTOMER  SATISFACTION 
SURVEYS  OFFICE  OF INSPECTOR  GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 
amended, is to protect the integrity of the Department of Health and Human Services (HHS) 
programs as well as the health and welfare of beneficiaries served by those programs.  This 
statutory mission is carried out through a nationwide network of audits, investigations, and 
inspectionsconductedby three OIGoperatingcomponents: the Officeof AuditServices,the 
Office of Investigations, and the Office of Evaluation and Inspections. The OIG also informs the 
Secretaryof HHSofprogram andmanagementproblemsand recommendscoursesto correct them. 
OFFICE  OF AUDIT  SERVICES 
The OIGSOfficeof AuditServices(OAS)providesall auditingservicesfor HHS, either by 
conducting audits with its own audit resources or by overseeing audit work done by others.  Audits 
examine the performance of HHS programs and/or its grantees and contractors in carrying out 
their respectiveresponsibilitiesand are intendedto provide independentassessmentsof HHS 
programs and operations in order to reduce waste, abuse, and mismanagementand to promote 
economy and efficiency throughout the Department. 
OFFICE  OF INVESTIGATIONS 
The OIGSOfficeof Investigations(01) conductscriminal, civil, andadrnhistrative investigations 
of allegations of wrongdoing in HHS programs or to HHS beneficiaries and of unjust enrichment 
by providers.  The investigative efforts of 01 lead to criminal convictions, adrnhistrative 
sanctions, or civil money penalties.  The 01 also oversees State Medicaid fraud control units which 
investigate and prosecute fraud and patient abuse in the Medicaid program. 
OFFICE  OF EVALUATION  AND INSPECTIONS 
The OIGSOffice of Evaluation and Inspections (OEI) conducts short-term management and 
program evaluations(calledinspections)that focuson issuesof concernto the Department,the 
Congress, and the public.  The findingsand recommendationscontainedin these inspectionreports 
generate rapid, accurate, and up-to-date information on the efficiency, vulnerability, and 
effectivenessofdepartmentalprograms.  This reportwasprepared in the NewYork regionaloffice 
under the direction of Alan S. Meyer, Ph.D., Regional Inspector General.  Project staff included: 
REGION II  HEADQUARTERS 
Demetra Arapakos, Project Leader  Jennifer Antico 
Nancy Harrison  Hugh Hetzer 
Brian Ritchie 
Barbara Tedesco 
For additional copies of this report, please contact the New York regional office at (212) 264-
1998.  Department  of  Health  and  Human  Services 
OFFICE  OF 
INSPECTOR  GENERAL 
HMO  CUSTOMER  SATISFACTION

SURVEYS

# sERVICq 
J+  G. 
*+  7 
GIBBS  BROWN
g  JUN13 
u Inspector  General 
2
% 
%++,< 
MARCH  1995 
2 
*daa OEI-02-94-O0360  EXECUTIVE  SUMMARY

To assess how Medicare  health maintenance  organizations  (HMOS) are conducting 
customer  satisfaction  surveys and how they are utilizing  the results  of these surveys. 
BACKGROUND 
In various  staff meetings,  the Office of Managed  Care  in the Health  Care Financing 
Administration  (HCFA)  asked the Office of Inspector  General  (OIG) to survey how 
Medicare  HMOS are measuring  customer  satisfaction,  particularly  of Medicare 
beneficiaries,  and using the resulting  data.  The HCFA  requested  this study in order to 
better  ascertain  how active its role should be in surveying  Medicare  HMO enrollees  and 
how the surveys HMOS are conducting  can be of use to HCFA  in its monitoring  efforts. 
We selected  a stratified  random  sample of 95 HMO risk and cost contracts  out of the 
universe  of 185 such contracts  with Medicare  beneficiaries  enrolled  as of February  1, 
1995.  We sent them a mail questionnaire  regarding  their customer  satisfaction  survey 
procedures  and their use of survey results.  We also requested  copies of their  survey 
instruments,  which we analyzed  for content  and format.  We received  72 completed 
questionnaires  and 63 survey  instruments. 
FINDINGS 
Virtually  All Risk and Cost HMOS Conduct Customer  Satisfaction  Surveys 
All but one of our respondent  HMOS (99 percent)  conduct  general  customer  satisfaction 
surveys. 
However,  Most HMOS Do Not Target Their Medicare  Members 
More than half of the HMOS (55 percent)  have never  conducted  a customer  satisfaction 
survey of their Medicare  members  only.  Furthermore,  almost all of these (97 percent) 
also do not include questions  specific to Medicare  members  on their general  surveys. 
More than one-third  of all HMOS (39 percent)  do not know the satisfaction  rate of their 
Medicare  members  for their last general  survey,  and most (65 percent)  do not know the 
Medicare  response  rate.  However,  the Medicare  specific data which is available  shows 
that Medicare  members  have high satisfaction  rates. 
HMO  Customer  Satisfaction  Survey Instruments  and Procedures  Lack Uniformity 
The satisfaction  survey instruments  used by HMOS vary widely in their format  and 
content;  in fact,  no two are the same.  These instruments  differ  in their 
comprehensiveness  and in the rating  scales and satisfaction  questions used.  The survey 
i  procedures  HMOsuse  also vary.  This lack ofuniformity  in HMO surveys renders 
comparisons  between HMOS difficult,  if not impossible,  when assessing Medicare 
beneficiaries  satisfaction  with their plans. 
While in Many Ways Basically  Sound,  Technical  Weaknesses  in Many HMO  Surveys 
May Mask Problems  and Inflate Satisfaction  With Managed  Care Plans 
Many HMOS appear to be following  sound survey principles  regarding  sampling  and 
increasing  response  rates.  Most are also using survey  instruments  containing  clearly 
worded  and focused  questions  (98 percent)  and covering  a broad  range  of satisfaction 
dimensions  (71 percent.)  However,  many HMO survey  instruments  do contain 
weaknesses  which may bias,  to some degree,  survey results.  Most significantly,  more 
than half (58 percent)  include no questions about problems  with or complaints  about health 
plan services,  and twenty-nine  percent use survey instruments  which include an 
unbalanced  five-point  rating  scale of three positive,  one neutral  and only one negative 
response  category.  Most HMOS are also lacking mail follow-up  procedures  (21 of 32 
which use mail surveys)  and are not conducting  non-respondent  analyses  (87 percent.) 
One-third  of HMOS scored  adequate on an index of instrument  adequacy,  and less than 
half (44 percent)  scored adequate  on an index of procedure  adequacy. 
HMOS Use Their Survey Results as Much for Marketing  as for Quality Improvement 
Nearly three-fourths  of HMOS (74 percent)  say they use the results  of their customer 
satisfaction  surveys to market  themselves  to potential  new members.  A similar  number 
(76 percent)  report  also using their customer  satisfaction  data to develop  quality 
improvement  or corrective  action plans.  A majority  of HMOS share their survey  findings 
with physicians  (76 percent)  and with plan members  (67 percent.) 
IMPLICATIONS  FOR MEDICARE 
We believe  the usefulness  of customer  satisfaction  surveys as currently  conducted  by 
HMOS is substantially  reduced  by their lack of uniformity,  limited focus on Medicare 
beneficiaries  and technical  weaknesses.  Therefore,  if HCFA  seeks data on the satisfaction 
of Medicare  beneficiaries  with managed  care,  we believe  it can not rely upon industry 
surveys as they are now conducted.  The HCFA  may want to consider  alternative 
approaches  to measuring  Medicare  client  satisfaction  with managed  care,  such as 
conducting  its own surveys or requiring  HMOS to periodically  survey their Medicare 
members  with a standardized  instrument  and comparable  procedures. 
The Office of Inspector  General  is planning  further  work surveying  Medicare  beneficiaries 
about their experiences  with HMOS.  We will once again conduct  a survey of Medicare 
HMO members  similar  to one already  completed.  We are also working  on a technical 
assistance  report  which will provide  HCFA  with a more detailed  discussion  of our 
methodology  for conducting  beneficiary  surveys and will identify useful  survey techniques 
and methods based on our prior  experience. 
ii  COMMENTS 
Wereceived  favorable  comment.s from HCFAontie  drafi repoti,  expressing  their 
appreciation  for the timeliness  and significance  of our study,  In particular,  HCFA  states 
that this report  will be a major  factor  in influencing  its decision to develop  its own 
beneficiary  satisfaction  survey capability.  The actual comments  received  are included  in 
Appendix  C. 
... 
m  TABLE  OF  CONTENTS 
PAGE 
EXECUTIVE  SUMMARY 
INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

FINDINGS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...5

  Virtually All HMOs Conduct General Surveys  . . . . . . . . . . . . . . . . . . . . . ..5

  Most HMOs Do NotTargetMedicare  Members  . . . . . . . . . . . . . . . . . . . ...5

  HMO Surveys Lack Uniformity.  . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...6

  Technical  Weaknesses  May Mask Problems  and Inflate Satisfaction  . . . . . . . . . . .7

  HMOs Use Survey Results as Much for Marketing  as for Quality  Improvement.  . . .9

IMPLICATIONS  FOR  MEDICARE  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

APPENDICES 
A:  Statistical Procedures  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..  A-1

B:  Indexes of Survey Adequacy  .,.  . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. B-l
 INTRODUCTION 
PURPOSE 
To assess how Medicare  health maintenance  organizations  (HMOS) are conducting 
customer  satisfaction  surveys and how they are utilizing  the results of these surveys. 
BACKGROUND 
In various  staff meetings,  the Office of Managed  Care  in the Health  Care  Financing 
Administration  (HCFA)  asked the Office of Inspector  General  (OIG) to survey how 
Medicare  HMOS are measuring  customer  satisfaction  and using the resulting  data.  The 
OIG was asked to pay particular  attention to what the managed  care industry  is doing to 
measure  the satisfaction  of its Medicare  members.  The HCFA  requested  this study in 
order to better  ascertain  how active its role should be in surveying  Medicare  HMO 
enrollees  and how the surveys HMOS are conducting  can be of use to HCFA  in its 
monitoring  efforts. 
This report  follows earlier  OIG reports  on Medicare  managed  care.  These  included  one 
entitled  Beneficiary  Perspectives  of Medicare  Risk HMOS which surveyed  beneficiaries 
about their experiences  with HMOS.  This report  found that while risk HMOS provide 
adequate  service  access for most of their Medicare  enrollees,  some serious problems 
remained  with enrollment  procedures  and service  access and disenrollees  were more likely 
than enrollees  to have experienced  problems  with HMO services.  A further  report  on 
Medicare  Risk HMOS:  Beneficiary  Enrollment  and Service Access Problems  looked 
more closely at HMO-level  data to identify the distribution  of these problems  and found 
that they existed in varying  degrees  of intensity among HMOS and more  frequently  with 
disenrollees  than with enrollees.  Another  report,  entitled  Medicare  Risk HMO 
Performance  Indicators,  found that HMO disenrollment  rates,  along with customer 
satisfaction  surveys,  appear to be useful managed  care performance  indicators. 
l%e Industry 
Health maintenance  organizations  are a form of managed  care in which a patient  selects a 
primary  care physician  from a group of approved  plan providers  to act as the patients 
first point of contact  within the health care system.  This physician  must authorize  any 
specialist,  hospital  or other type of care the patient receives.  According  to industry 
estimates,  approximately  50 million  individuals  in the United  States are enrolled  in one of 
the 550 HMOS across the country, 
An HMO can pay its physicians  in different  ways.  A primary  care physician  or specialist 
can be paid on a cavitation  basis,  in which he or she is paid one monthly  amount per each 
patient regardless  of how much care that patient  receives.  Physicians  can also be paid on 
a fee-for-service  basis.  Some HMOS use a combination  of these different  payment 
methods. 
1  Medicare  HMOS 
Medicare  beneficiaries  have the option of receiving  their health care from an HMO 
approved  by HCFA.  Once approved,  the HMO generally  applies for a risk or cost 
Medicare  contract.  In a risk contract,  the HMO provides  the full Medicare  benefit 
package  andispaid  onaprospective  percapita  basis,  inwhich  itisrequired  to absorb any 
financial  losses butispemitied  toremin  any fimncial  savings.  Under  such a contract, 
payment  ismadeon  aprepaid  cavitation  basis with no retroactive  adjustment.  The HCFA 
encourages  HMOS to apply for risk contracts.  In a cost contract,  the HMO also provides 
the full Medicare  benefit package but is paid on a reasonable  cost basis.  An HMO can 
also serve Medicare  patients  through  a health care prepayment  plan agreement  or as a 
demonstration  project. 
The number  of Medicare  risk and cost contracts  continues  to grow.  As of February  1, 
1995,  157 plans had risk contracts  and 30 had cost contracts  with HCFA,  with a total 
enrollment  of approximately  2.5 Medicare  million beneficiaries.  This is an increase  from 
July of last year,  when there  were  136 risk HMOS and 27 cost HMOS.  In February  1993, 
only 87 HMOS had risk contracts  to serve Medicare  beneficiaries. 
While only about seven percent  of the Medicare  population  nationwide  is enrolled  in 
HMOS, the geographical  distribution  of  enrollment  varies  widely.  The distribution  of 
tlis 
Medicare  HMO enrollees  is concentrated  in four States:  California,  New York,  Florida, 
and Arizona.  A few States have no Medicare  beneficiaries  enrolled  in managed  care 
programs. 
Section 42 CFR 417. 107(h) of the regulations  require  federally  qualified  HMOS with 
Medicare  enrollees  to implement  ongoing quality assurance  programs.  These programs 
must have the following basic components:  a quality assurance  methodology,  a peer 
review process,  systematic  data collection  of performance  and patient  results,  and 
remedial  action procedures. 
As part of the systematic  data collection  requirement  listed above,  HMOS are required  to 
share their data collection  results with their providers  of care and institute  any needed 
changes based on these results.  Data can be collected  from any of several different 
sources,  including member  satisfaction  surveys.  These surveys,  however,  are not required 
by law. 
Customer Satisfaction  Initiatives 
Several efforts have been under  way in the managed  care  industry to develop 
methodologies  and instruments  for measuring  and reporting  performance  ratings,  including 
customer  satisfaction,  in managed  health care.  These  initiatives  have primarily  been 
motivated  by an interest  in the industry and among health care consumers  in establishing 
standard measures  by both individuals  and employers  can compare  and contrast  different 
health plans.  Few of these initiatives,  however,  are specifically  aimed at the Medicare 
population. 
2  Two industry groups  are particularly  active in managed  care quality and customer 
satisfaction  issues.  The National  Committee  for Quality Assurance  (NCQA)  is a voluntary 
private  accreditation  agency active in setting and enforcing  HMO quality standards.  In 
November  1993, it developed  the Health Plan Employer  Data and Information  Set 
(HEDIS),  which defines performance  measures  to evaluate,  among other things,  a plans 
quality of care,  member  access to care,  and member  satisfaction.  The NCQA publishes 
the results  of its accreditation  reviews  nationwide.  Also, the Group Health  Association  of 
America  (GHAA),  a managed  care industry  group to which most HMOS belong,  has 
developed  a model consumer  satisfaction  questionnaire  which is available  to its members. 
METHODOLOGY 
In conducting  this inspection,  we selected a stratified  random  sample out of the universe 
of 185 HMO risk and cost contracts  with Medicare  beneficiaries  enrolled  as of February 
1, 1995 (nine additional  HMO risk and cost contracts  were dropped  from the universe 
because they had no Medicare  enrollees  at the time the sample was drawn.)  These HMO 
contracts  were stratified  into three groups  of high,  medium  and low Medicare  enrollment, 
based on the number  of Medicare  beneficiaries  in each.  The HMOS in the high stratum 
have Medicare  enrollments  of larger  than 42,550,  while those in the medium  and low 
strata have between 42,500  and 1000 and less than  1000 Medicare  members  respectively, 
We purposely  contacted  all of the contracts  in the high stratum.  See Appendix  A for a 
more detailed  explanation  of strata  selection. 
We selected 95 HMO risk and cost contracts  for the fiml  sample:  all 13 from the first 
stratum,  70 from the second stratum,  and 12 from the third  stratum.  Medicare  enrollment 
in the  13 high stratum contracts  accounts  for 51 percent  of all Medicare  enrollment 
nationwide  in managed  care.  Forty-seven  different  HMOS across the country  hold these 
95 contracts,  Eighty-two  of the 95 are risk contracts,  and the remaining  13 are cost 
contracts. 
We sent all 95 a mail questionnaire  which requested  information  about their customer 
satisfaction  survey procedures  and their use of survey results.  We also requested  copies 
of the HMOS survey instruments.  After allowing  six weeks for data collection,  during 
which time we conducted  a second mailing to non-respondents,  we achieved  an overall 
response  rate of 76 percent  for the questionnaires.  We also achieved  a 66 percent  overall 
response  rate for the survey  instruments  after making  a minimum  of two follow-up 
telephone  calls to HMOS who did not initially  send us their  instruments.  For the mail 
questionnaires,  we achieved response  rates of 100 percent  for the high stratum,  77 percent 
for the medium  stratum,  and 42 percent  for the low stratum.  For the survey instruments, 
we achieved response  rates of 100, 64 and 42 percent  for each of the three  strata 
respectively.  All differences  reported  between  strata are statistically  significant  at the 95 
percent  conildence  level. 
Differing  response  rates among strata suggest the possibility  of non-response  bias.  While 
we did use Chi-square  to test for such bias,  due to our relatively  small sample size and 
3
 resulting  small cell sizes in the two-variable  tables,  it was not a valid test.  Therefore,  we 
acknowledge  the possibility  of non-response  bias but are not able to quanti@ it. 
To assess HMO customer  satisfaction  survey procedures,  we reviewed  the returned  mail 
questionnaires  to determine,  among other things,  their sampling methods,  follow-up 
procedures,  and their means of determining  response  and satisfaction  rates.  Our findings 
on survey procedures  are based on the HMOS answers to these questionnaires.  See 
Appendix  A for contldence  intervals  of fifteen key questions. 
To assess HMO survey  instruments,  we developed  a detailed review  sheet which we used 
to systematically  evaluate  each instrument  for both form and content.  This review  sheet 
included  assessments  of each instruments  length,  dimensions  of satisfaction  measured, 
scales and format used,  clarity  of instructions  and questions,  and user friendliness. 
We also constructed  two indexes of survey adequacy  - one for instruments  and the other 
for procedures.  We used data from the instrument  review  sheets discussed  above to 
construct  the index of instrument  adequacy.  This index was based on three  key variables: 
comprehensiveness  of satisfaction  dimensions,  balanced  response  categories,  and problem 
specific questions.  The index of procedure  adequacy was based on four variables:  level of 
confidence  sought,  frequency  of follow-up efforts,  conducting  a non-respondent  analysis, 
and use of balanced  criteria  for determining  overall  satisfaction.  For these four variables, 
we used indicators  from the questionnaires  returned  by HMOS.  In each of the two 
indexes,  we gave the variables  a subscore,  which were then combined  to give total scores 
for instrument  adequacy  and procedure  adequacy  respectively.  The indexes of survey 
adequacy  are explained  in greater  detail in Appendix  B. 
This inspection  was conducted  in accordance  with the Quality  Standards  for Inspections 
issued by the Presidents  Council  on Integrity  and Efficiency. 
4
 FINDINGS 
VIRTUALLY  ALL RISK AND COST HMOS CONDUCT  CUSTOMER 
SATISFACTION  SURVEYS 
General Membership  S@sfaction Surveys 
All but one of the respondent  HMOS (99 percent)  conduct  general  customer  satisfaction 
surveys.  Almost all (95 percent)  consider  these general  surveys to be very or somewhat 
useful.  A majority  of HMOS (60 percent)  conduct their surveys at least once a year, 
while thirteen  percent  survey their members  twice a year.  Most of the remaining  28 
percent  conduct  satisfaction  surveys on an ongoing basis. 
Nearly three-fourths  of HMOS occasionally  (24 percent)  or always (49 percent)  use a 
professional  research  agency to conduct their customer  satisfaction  surveys,  with a total of 
37 different  research  fms  cited by the responding  HMOS.  All of the HMOS with high 
Medicare  enrollment  hire professional  research  firms,  as compared  to 70 percent  of 
HMOS with medium  and low Medicare  enrollments. 
Other ~pes  of Surveys 
A majority  of HMOS are also conducting  other types of surveys.  These  include  surveys 
of disenrollees  (92 percent),  on the functional  or health status of their members  (58 
percent),  and of the working  aged (53 percent.)  Fifty-nine  percent  of HMOS also survey 
the physicians  who work for them about their  satisfaction. 
HOWEVER,  MOST WOS  DO  NOT  TARGET  THEIR  MEDICARE  MEMBERS 
Medicare  Stisfaction  Surveys 
While HMOS conduct  satisfaction  surveys of their general  memberships,  they pay less 
specific attention to their Medicare  members.  More than half of the HMOS (54 percent) 
have never conducted  a customer  satisfaction  survey of their Medicare  members  only;  in 
fact, eight percent  of these survey only their commercial,  non-Medicare,  members  on 
satisfaction.  Seven of the HMOS who have not yet conducted  a Medicare  only survey did 
volunteer,  however,  that they are planning  to do so within the next year.  Almost all of 
the HMOS (97 percent)  which do not conduct Medicare  only surveys also do not include 
questions  specific to Medicare  members  on their general  surveys;  the few who do ask only 
a limited number  of Medicare  questions. 
While  most HMOS have the potential  to identify Medicare  members  in their general 
surveys by identifying  the respondents  age,  for the most part they are not extracting 
Medicare  specific data.  More than one-third  of all HMOS (39 percent)  do not know the 
satisfaction  rate of their Medicare  members  for their  last general  survey,  and most (65 
percent)  do not know the Medicare  response  rate.  Nevertheless,  whether  or not they 
5  survey Medicare  enrollees,  more than half of the HMOS (58 percent)  believe  it is easier to 
survey Medicare  members  than it is to survey non-Medicare  members,  primarily  because 
the former  are more responsive,  have more time,  and are easier to reach. 
The Medicare  data which is available  shows that Medicare  HMO members  have high 
satisfaction  rates.  All of the HMOS with Medicare  data report  overall  satisfaction  rates of 
75 percent  or higher  for their Medicare  members.  Similarly,  virtually  all (99 percent) 
report  the same 75 percent  or higher  satisfaction  rate for all members. 
HMO CUSTOMER  SATISFACTION  SURVEY INSTRUMENTS  AND 
PROCEDURES  LACK UNIFORMITY 
The lack of uniformity  in HMO survey instruments  and procedures  renders  comparisons 
between  HMOS difficult,  if not impossible,  when assessing Medicare  beneficiaries 
satisfaction  with their plans. 
Satisfaction  Survey Instruments 
The survey instruments  used by HMOS vary widely in their format  and content.  No two 
risk and cost HMOS use the same instrument.  Of our sample survey  instruments,  none 
use an exact duplication  of the GHAA satisfaction  survey instrument,  although  almost half 
(48 percent)  use some exact or similar headings,  questions and rating  scales from that 
survey.  The HMO instruments  differ  significantly  in their length,  ranging  in size from  1 
to 45 pages and including  from between  9 to 159 different  questions. 
Rating scales and overall  satisfaction  questions on survey instruments  are also inconsistent 
across HMOS.  Our review  of HMOS instruments  identified  26 different  scales used for 
responses,  ranging  from a simple two-point  scale of yes or no, to a ten-point  scale 
covering  a range of satisfaction  levels.  Fourteen  different  types of questions  are used to 
measure  overall  satisfaction,  including questions which ask about satisfaction  with medical 
care,  most recent  visit,  quality of service,  health plan and a particular  medical  center. 
Three-fourths  of HMOS, however,  use a question about satisfaction  with their plan to 
measure  overall  satisfaction. 
Stisfaction  Survey Procedures 
Sampling procedures  also differ.  Fourteen  percent  of HMOS survey their entire 
membership.  Of the 86 percent  who survey a sample of their membership,  sample sizes 
vary from  100 to over 5000.  The size and type of sample used varies  according  to the 
purpose  of each HMOs  survey.  While most (51 percent)  use a simple random  sample, 
another 41 percent  use a stratified  random  sample,  and eight percent  a purposive  sample. 
Forty-nine  percent  of the HMOS select their  sample from a universe  of one particular 
subgroup,  as defined,  for example,  by a minimum  length of membership,  while most of 
the remaining  sample from the universe  of their entire membership. 
6
 Furthermore,  other survey procedures  used by HMOS vary.  Forty-four  percent  administer 
their customer  satisfaction  survey by telephone  only,  16 percent  administer  it by telephone 
and mail,  and 37 percent  by mail only.  The remaining  generally  use an in-office  self-
administered  questionnaire  in combination  with mail or telephone.  Excluding  those which 
conduct  ongoing surveys,  sixty-two percent  of HMOS stop data collection  after ten weeks 
and the remaining  third continue  collecting  data for longer than ten weeks. 
WHILE  IN MANY WAYS BASICALLY  SOUND,  TECHNICAL  WEAKNESSES  IN 
MANY HMO SURVEYS MAY MASK PROBLEMS  AND INFLATE 
SATISFACTION  WITH MANAGED  CARE PLANS 
HMO Survey Strengths 
Many HMOS appear to be following  sound survey principles  regarding  sampling,  efforts 
to increase  response  rates by telephone,  and instrumentation.  A majority  (71 percent) 
were seeking conildence  intervals  of 95 percent  or higher  in designing  their  sample size. 
Close to half (44 percent)  used sample sizes of over  1000 members,  which,  in 
combination  with their generally  high response  rates,  should have ensured  a good level of 
precision  in their surveys.  Additionally,  most HMOS who use telephone  surveys make a 
strong effort  to maximize  their response  rates.  Almost all of these (94 percent)  try to 
contact members  at least three times before  considering  them to be non-respondents. 
Furthermore,  our review  of HMO survey instruments  revealed  several positive  features. 
A great majority  of the instruments  generally  include questions which are specific  (92 
percent),  clearly  worded  (98 percent)  and focused on only one thought  at a time  (98 
percent.)  The instruments  also cover a broad  range of satisfaction  dimensions.  The most 
common  include ability to provide  prompt  service  (96 percent),  overall  satisfaction  with 
services  (87 percent),  staff courtesy  (84 percent),  access to services  (82 percent), 
physician  communication  (81 percent),  and physician  competence  (77 percent.)  Of the 
seventeen possible dimensions  of satisfaction  we looked for in the HMOS instruments,  a 
majority  (71 percent)  include at least ten. 
HMO Survey Instrument  Weaknesses 
However,  our instrument  review  also revealed  some weaknesses  in the survey  instruments 
which may bias,  to some degree,  survey results.  More than half of the instruments  (58 
percent)  include no questions about problems  with or complaints  about health plan 
services.  Of the 42 percent  which do include these topics,  most ask only one or two 
questions.  For the most part,  the questionnaires  ask only if a member  has ever had a 
problem  with or complaint  about their care,  and how satisfied he or she was with its 
resolution.  The OIG report  referenced  earlier  on beneficiary  perspectives  of risk HMOS 
included  several  questions in its beneficiary  survey specific to problems  with HMO 
services.  These questions were important  in understanding,  among other things,  reasons 
for dissatisfaction  and disenrollment  with HMOS.  Half of the instruments  (51 percent)  do 
not ask respondents  for their suggestions  for improving  the HMO. 
7
 Other deficiencies  noted in the HMOS instruments  may compromise  their objectivity. 
Forty-six  percent  use at least one unbalanced  rating  scale.  The most common  of these,  a 
five-point  rating  scale of three positive,  one neutral  and only one negative  response 
category,  is used by twenty-nine  percent  of the HMOS.  Furthermore,  of the quarter 
which use agree or disagree  statements,  almost all include only positive  statements  for 
respondents  to respond  to.  While this has the advantage  of ensuring  greater  ease for the 
respondent,  it also has the disadvantage  of possibly resulting  in more positive  ratings. 
Other weaknesses  noted include few or unclear  interviewer  instructions  on telephone 
survey instruments  (24 percent),  confusing  or unclear  questionnaire  formats  (20 percent), 
and inconsistent  rating  scales (20 percent)  and repetitive  questions within an instrument 
(15 percent.) 
We also rated each survey  instrument  to determine  its level of user friendliness.  In doing 
this,  we looked for several  qualities,  including clarity  of format,  simplicity  of directions 
and questions,  and overall  attractiveness.  While most surveys  (73 percent)  were rated 
user friendly,  only eleven percent  were rated  very user friendly  and 16 percent  were 
deemed not user  friendly.  A majority  (80 percent)  of those rated user friendly  are mail 
surveys,  while most (70 percent)  rated  not user friendly  for either  the respondent  or the 
interviewer  are telephone  surveys. 
HMO Survey Procedure  Weaknesses 
Most HMOS are also lacking certain  survey procedures  which,  if used,  may increase 
survey accuracy.  Twenty-nine  of 32 sample HMOS which use mail surveys do not try to 
contact members  who do not return  their original  questionnaires;  just  one-third  send non-
respondents  a second copy.  Additionally,  a large majority  (87 percent)  have never 
conducted  a non-respondent  analysis,  which would enable them to know if non-
respondents  differed  in any way from respondents,  thus alerting  them to possible bias in 
the survey findings.  Finally,  less than half of the HMOS (43 percent)  have ever 
conducted  a bilingual  survey.  All of these  conducted  surveys  in Spanish,  and 
HMC)S  have 
many are based in States with a large  Spanish-speaking  population.  The lack of a 
bilingual  survey may be a survey weakness  in those geographical  areas with large  numbers 
of non-English  speaking clients;  without them,  the experiences  and satisfaction  of these 
clients can not be fully comprehended. 
Indexes  of Survey Adequacy 
As described  in our methodology  and explained  in Appendix  B, we constructed  two 
indexes of survey adequacy  - one for instruments  and the other for procedures.  One-third 
of the HMOS instruments  (32 percent)  scored adequate  on our survey  instrument  index, 
while another  28 percent  scored somewhat  adequate.  Of the remaining  instruments,  24 
percent  scored  somewhat  inadequate  and 16 percent  inadequate. 
The HMOS were also scored on the adequacy  of their  survey procedures.  On this index, 
less than one-half  (44 percent)  scored adequate,  and thirty-one  percent  scored  somewhat 
8

- adequate.  Another  ten percent  of the HMOS scored somewhat  inadequate,  while  15 
percent  scored inadequate. 
HMOS USE THEIR SURVEY RESULTS  AS MUCH FOR MARKETING  AS FOR 
QUALITY  IMPROVEMENT 
Marketing 
A majority  of HMOS are using their  satisfaction  surveys in their marketing  plans.  Nearly 
three-fourths  (74 percent)  say they use the results  of their customer  satisfaction  surveys to 
attract potential  new members.  Most of this marketing  consists  of reporting  survey data 
in speeches or other oral presentations,  and in pamphlets  or brochures.  Seventeen percent 
use their survey findings for advertising  purposes.  The HMOS which use their  survey 
results  for marketing  have higher  overall  satisfaction  rates than those which do not market 
their survey results;  46 percent  of the former  report  overall  satisfaction  rates of 75 percent 
or more,  while 20 percent  of the latter report  the same overall  satisfaction  rates. 
Quality Improvement 
A similar number  of HMOS utilize  survey results  for quality improvement  purposes. 
Three-fourths  (76 percent)  report  using their customer  satisfaction  data to develop 
improvement  or corrective  action plans.  One-third  (34 percent)  use the results  for 
tracking  performance  and developing  strategic  goals.  Almost all (97 percent)  include their 
satisfaction  surveys in their Medicare  quality assurance  programs. 
Most HMOS also share their  survey results with employees  and members.  A majority 
distribute  their  survey findings to physicians  (76 percent)  and to plan customers  (67 
percent.)  In fact,  more than one-third  (36 percent)  use these results to determine  all or 
part of their physician  reimbursement.  Of those HMOS who conduct  satisfaction  surveys 
of their physicians,  a few (16 percent)  compare  these results to those of their member 
satisfaction  survey results. 
Differences  Between  HMOS 
The HMOS use their survey results  somewhat differently  depending  on the size of their 
Medicare  enrollment.  Almost all of the HMOS with high Medicare  enrollment  (92 
percent)  use their  survey data for quality improvement  purposes,  compared  to 75 percent 
of HMOS with medium  and low Medicare  enrollments.  Marketing  of survey results  is 
conducted  by 58 percent  of high enrollment  HMOS, in contrast  to 75 percent  of medium 
and low enrollment  HMOS.  A comparison  of the instruments  used by high Medicare 
enrollment  HMOS with those used by medium  and low Medicare  enrollment  HMOS 
reveals  some important  differences  in this regard.  For example,  more than half of the 
former  (69 percent)  have questions which ask members  about problems  or complaints  with 
their health plans,  as compared  to just  forty percent  of the latter,  Also,  HMOS with a 
large number  of Medicare  enrollees  are more likely to ask for suggestions  for 
improvement  on their questionnaires  than are HMOS with fewer Medicare  enrollees. 
9  We believe  the usefulness  of customer  satisfaction  surveys as currently  conducted  by 
HMOS is substantially  reduced  by their lack of uniformity,  limited focus on Medicare 
beneficiaries  and technical  weaknesses.  Therefore,  if HCFA  seeks data on the satisfaction 
of Medicare  beneficiaries  with managed  care,  we believe  it can not rely upon industry 
surveys as they are now conducted.  The HCFA  may want to consider  alternative 
approaches  to measuring  Medicare  client  satisfaction  with managed  care,  such as 
conducting  its own surveys or requiring  HMOS to periodically  survey their Medicare 
members  with a standardized  instrument  and comparable  procedures. 
The Office of Inspector  General  is planning  further  work surveying  Medicare  beneficiaries 
about their experiences  with HMOS.  We will once again conduct  a survey of Medicare 
HMO members  similar to one already  completed.  We are also working  on a technical 
assistance  report  which will provide  HCFA  with a more detailed  discussion  of our 
methodology  for conducting  beneficiary  surveys  and will identify useful  survey techniques 
and methods based on our prior  experience. 
COMMENTS 
We received  favorable  comments  from HCFA  on the draft report,  expressing  their 
appreciation  for the timeliness  and significance  of our study.  In particular,  HCFA  states 
that this report  will be a major  factor  in influencing  its decision to develop  its own 
beneficiary  satisfaction  survey capability.  The actual comments  received  are included  in 
Appendix  C. 
10
 APPENDIX  A 
STATISTICAL  PROCEDURES 
z.  SAMPLE  SELECTION 
Our original  universe  of HMO Medicare  contracts  consisted  of 193 risk and cost contracts 
which had been awarded  as of February  1, 1995.  However,  we eliminated  eight of these 
contracts  which had no Medicare  enrollees  at the time our sample was drawn.  This left us 
with a universe  of 185 risk and cost contracts. 
We stratified  these  185 contracts  into three  groups - large,  medium,  and small - based  on 
the number  of Medicare  enrollees  in each HMO contract.  Just  13 large HMO contracts 
accounted  for 51 percent  of all Medicare  managed  care enrollment.  We sampled these  13 
at a rate of 100 percent.  We sampled the medium  stratum HMOS at a rate of 54 percent 
and the low stratum HMOS at a rate of 28 percent. 
The sample size of 95 was based on the assumption  of a 75 percent  response  rate.  The 95 
were stratified  as follows: 
STRATA  # OF MEDICARE  ENROLLEES  UNIVERSE  SAMPLE 
1 (large)  >  42500  13  13 
2 (medium)  1000-42500  130  70 
3 (low)  <1000  42  12 
TOTAL  185  95 
1
 II.  CONFIDENCE INTERVWS  FOR 15 KEY QUESTIONS 
We calculated  conildence  intervals  for fifteen key questions.  The response  estimate  and 
95% confidence  interval  is given for each of the following: 
From the mail cwestionnaire 
1.	 Have you ever conducted  a customer  satisfaction  survey of any kind? 
Yes response  estimate:  99% 
Lower  interval:  97% 
Upper  interval:  100% 
2.	 Have you ever conducted  a customer  satisfaction  survey of your Medicare 
enrollees  only? 
No response  estimate:  55% 
Lower  interval:  44% 
Upper  interval:  66% 
3.	 How many times did you try to contact members  who did not return  the mail 
questionnaire  before  finally considering  them to be non-respondents? 
No contact  estimate:  65% 
Lower  interval:  49% 
Upper  interval:  80% 
4.	 How many times did you try to reach members  by telephone  before  finally 
considering  them to be non-respondents? 
3 or more times estimate:  94% 
Lower  interval:  89% 
Upper  interval:  98% 
5.	 Did your survey instrument  include any questions which were asked only of 
Medicare  enrollees? 
No estimate:  97% 
Lower  interval:  93% 
Upper  interval:  100% 
6.	 What conildence  level were you seeking? 
95% or higher  estimate:  71% 
Lower  interval:  57% 
A-2
 Upper  interval:  85%

7,  For your Medicare  enrollees  only,  what response  rate did you achieve  in your last

customer  satisfaction  survey?

Dont Know estimate:  65%

Lower  interval:  54%

Upper  interval:  76%

8.  What percentage  of your Medicare  members  only did you fiid  to be satisfied

overall?

Dont Know estimate:  39%

Lower  interval:  30%

Upper  interval:  49%

9.  Did you use the results  of your customer  satisfaction  survey to develop  a quality

improvement  or corrective  action plan?

Yes estimate:  76%

Lower  interval:  67%

Upper  interval:  86%

10.  Did you share the results of your customer  satisfaction  survey with your members?

Yes estimate:  67%

Lower  interval:  57%

Upper  interval:  77%

11.  Did you share the results  of your customer  satisfaction  survey with the physicians

who work for your HMO?

Yes estimate:  76%

Lower  interval:  66%

Upper  interval:  87%

12.  Did you use the results  of your customer  satisfaction  survey to market  the HMO to

potential  new members?

Yes estimate:  73%

Lower  interval:  65%

Upper interval:  83%

From the survey instrument  review  sheet 
13.	 Generally,  are  [instrument]  questions worded clearly  and focused on one thought? 
Yes estimate:  98% 
Lower  interval:  96% 
Upper  interval:  100% 
14.	 Did the instrument  use a rating  scale of poor,  fair,  good,  very good,  and excellent? 
Yes estimate:  29% 
A-3
 Lower  interval:  19% 
Upper  interval:  39% 
15.	 How many questions,  including subquestions,  ask about problems  with or 
complaints  about health plan services? 
No questions  estimate  58% 
Lower  interval:  45% 
Upper  interval:  71% 
APPENDIX  B 
INDEXES  OF SURVEY  ADEQUACY 
I.  INDEX  OF INSTRUMENT  ADEOUACY 
This index of instrument  adequacy  is based on the sum of scores achieved by each HMO 
on three  separate  variables: 
(1) comprehensiveness  of satisfaction  dimensions 
(2) balance  of response  categories 
(3) problem-specific  questions 
Scores on each variable  could range  from  +2  to -2 according  to the criteria  for each 
discussed below. 
(1)  Comprehensiveness  of S~sfaction  ~mensions 
The following  ten dimensions  of satisfaction  were  selected from a longer  list of 
such dimensions  which we looked for in the HMOS survey instruments:  patient 
access to general  or specialty care;  physician  or staff courtesy;  cost of care; 
coordination  or continuity  of care;  physician  competence;  choice of providers; 
communication  with physicians  or staffi office waiting time;  suggestions  for 
improving  HMOS; and, plans to stay with the plan. 
Based on the number  of these dimensions  included  in each HMOs  survey 
instrument,  the following  scores were used: 
No.  of dimensions  Score 
o-4  -2 
5  -1 
6-7  +1 
8-10  +2 
(2)  Bahznce of Response  Categories 
A-1
 Balance  of response  categories  refers  to whether  or not a rating  scale includes an 
even number  of positive  and negative  responses.  For example,  a scale with one 
positive,  one neutral,  and one negative  response  - good, fair,  poor - is balanced, 
while a scale with two positive,  one neutral,  and one negative  response  - very 
good,  good,  fair,  poor - is unbalanced.  Scores for this variable  were based on two 
items:  1) whether  response  categories  for overall  satisfaction  questions were 
balanced  or not; and 2) whether  response  categories  for all other questions  were all 
balanced,  almost all balanced,  only some balanced,  or all unbalanced.  Based on 
the extent to which HMOS used balanced  categories,  the following  scores were 
used: 
OVERALL  SATISFACTION  RESPONSE  CATEGORIES 
All  Not All  No Overall 
Balanced  Balanced  Question 
ALL OTHER RESPONSE

CATEGORIES

All Balanced
 +2  -1  +2 
Almost All Balanced
 +1  -1  -1 
Only Some Balanced
 -1  -2  -2 
None Balanced
 -2  -2  -2 
(3)  Problem-Specific  Questions 
Based on the number  of questions about potential  problems  or complaints  included 
in the HMOS survey  instruments,  the following  scores were used: 
# of Ouestions  Score 
o  -2 
1-4  +1 
5+  +2 
After scoring the HMOS on each of the three  variables,  we added the three  scores together 
to give a total instrument  adequacy  score for each HMO.  Scores ranged  from  +6  to -6, 
with a midpoint  of  +1.65.  The mean score is 1.1. 
The 63 HMO instruments  were grouped  into the following  four levels of adequacy based 
on their index scores: 
Level of Adeauacv  #HMOs  A  Score 
Adequate  20  32%  (+4  to  +6) 
Somewhat  adequate  13  28%  (+1  to  +3) 
Somewhat  inadequate  20  24%  ( o to -2) 
Inadequate  10  32%  (-3 to -6) 
II. INDEX  OF SURVEY PROCEDURE  ADEOUACY 
B-2
 This index of procedure  adequacy  is based on the sum of scores achieved by each HMO 
on four separate  variables: 
(1)  level of confidence  sought

(2)  frequency  of follow-up  efforts

(3)  conducting  a non-respondent  analysis

(4)  use of balanced  criteria  for determining  overall  satisfaction

Scores on each variable  could range from  +2  to -1 according  to the criteria  for each 
discussed below. 
(1)  Level  of Conj7dence Sought 
Based on the level of confidence  HMOS were  seeking when selecting their  survey 
samples,  the following  scores were used: 
Confidence  Interval  Score 
90% or higher  +1 
under  90%  -1 
Does not apply 
(did not use sample)  o 
(2)  Frequency  of Follow-up  Efforts 
The extent to which HMOS employed  follow-up procedures  with non-respondents 
was scored for each of three  survey modes:  mail,  telephone  and in-office.  The 
following  scores were used: 
A.  Mail: 
# times non-rest)ondents  are contacted:  Score 
2 or more times  +2 
1 time  +1 
No contact made  -1 
Does not apply (no mail survey)  o 
B.  Telephone: 
# times non-respondents  are contacted:  Score 
3 or more times  +2 
1 or 2 times  +1 
No contact made  -1 
Does not apply (no phone survey)  o 
c.  In-office  survey: 
B-3
 Follow-ut) mocedures  Score 
Follow-up  with refisals  +2 
No follow-up  with refhsals  -1 
Does not apply (no in-office  survey)  o 
None of the HMOS used all three modes.  However,  those which used both mail and 
telephone  to administer  their survey were given one combined  score for follow-up  of 
frequency  efforts.  These HMOS could not achieve a score greater  than  +2;  in this way, 
they were not given extra  credit for using two survey modes.  If they achieved  a negative 
score for one of the two modes,  it was subtracted  from any positive  scores to give a total 
score. 
(3)  Conducting  a Non-respondent  Analysis 
Based on whether  an HMO conducted  a non-respondent  analysis,  the following 
scores were used: 
Non-respondent  analvsis?  Score 
Yes  +1 
No  -1 
(4)  Use of Balanced  Criteria for  Determining  Overall Satlsfaction 
On the mail questionnaires,  we asked HMOS how they derived  an overall 
satisfaction  rating  for their members.  If they determined  this rating by taking only 
the positive  responses  from a scale which itself is balanced,  they used balanced 
criteria  to derive  overall  satisfaction.  Based on whether  an HMO used balanced 
criteria  in determining  an overall  satisfaction  rate,  the following  scores were used: 
Balanced Criteria?  Score 
Yes  +1 
No  -1 
As with the index of instrument  adequacy,  we added an HMOs  scores on these four 
variables  together  to give a total procedure  adequacy  score for each HMO.  Scores ranged 
from  +5  to -5, with a midpoint  of 2.  The mean score is 1.4. 
The 72 HMO questionnaires  on survey procedures  were grouped  into the following  four 
levels of adequacy  based on their  index scores: 
Level of Adequacy  #HMos  x  Score 
Adequate  32  44%  (+5  to  +3) 
Somewhat  adequate  22  31%  (+2  to  +1) 
Somewhat  inadequate  7  10%  ( o to -1) 
Inadequate  11  15%  (-2 to -5) 
B-4
 The  Adt]]tnijtrator 
Washington,  D.C.  -213201 
. 
DATE: 
To: 
FRONI: 
sw13JEc~: 
Ofke  Of ~spector  Gct\cral  Drafi Rcpofl:  Health  MaintenaL\ce 
@:tiations  @MOs)  CUs[oIner Satisfaction  Surveys, 
(OET:-02-94-00350) 
We  rw.kved  the ~.hove-referenced  &&  r,:po~ Wtich  assesse:; how HbIOS  are 
COndUCfig Custon:er  satisfaction  su~eys  ;tnd ho~~they  are ub~g  the reSuhS Of th~.$e 
S~eYS  The  s~d~  WrISconducted  in coctperation with  HCFA7S man:~:cd  care  o~cmtious 
and has provided  Inajor  insights  jnto  the need for ~  i.n&pendent  Offi,;eof  Ma.nagecl Care 
bcnefi;itiy  survey  capabi~~,  Ourdetauedcolmentsareattzchcdt 
Think  you  for the  opporh~fi$  to revie~v and co[~nel~t on tltis draft report. 
Attachment 
. 
. 
. 
.   
.. -  
..  
~&Mhumlkk  ,. 
TIIe OIG  study  has  provided  major  insight:; into the need  for ml independent oK~c~  of 
Managed  Care  lxn  :fici~  sumey  capabifil~  ~d  on the we  of issuesl:;wey  questions 
our pro~am  should  include. 
The  study  is nletilodolo5cal\y  sound  and th.e  sarn~le sizelresp,o ttn~ds e suraptpe olfalergde  beyn,o uti@e  sttoudy
assure  statistical  si;piflctice.  The  conclu;tons  are based  on 
data.  Find\ngs  of pdcdu  iniierest to us include: 
1;	 Wile  most  HMOS conduct  member  satisfaction  SWV%  most@  tcude 
Mcdicml:  members  in the surve)s.
, 
2.  Hh10  survey  instnwlents  lack Utifonfity  of approach  and dl:si~~. 
3,  Tecb~id  wea.hess:s  give ~  o~ef.~ated  mernbec  satisfaction  and 
SCnSC  of 
may  ma!;k PrOb@sI  ~ 
4.  Survey  c.ata are used  as much  fcr marketing  pUJ_POS~aSS for q~laliy 
imrmove:ment  purpo:;es.
. 
Ttti~ study  will  be  a major  factor  in influellckg  decision  to clevelo?  an mdependel~t
WC wouoru ld like to express  ow appreciation to 
beneficiary  satisfx:tiou  sines  capabui~. 
the OIG  for ad&e\  sing our operational  nctds  in such a timely  mumer.  \Ye hop~  fi~ 
futu.Tt will  intl~~dc stilar  joint  inte~agenty  cooperation. 
..