Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

Medicaid Managed Care: 

The Use of Surveys as a 

Beneficiary Protection Tool 

JUNE GIBBS BROWN 

Inspector General 

MAY 1997 

OEI-01-95-00280 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is 

to protect the integrity of the Department of Health and Human Services programs as well as 

the health and welfare of beneficiaries served by them. This statutory mission is carried out 

through a nationwide program of audits, investigations, evaluations, sanctions, and fraud alerts. 

The Inspector General informs the Secretary of program and management problems and 

recommends legislative, regulatory, and operational approaches to correct them. 

OFFICE OF EVALUATION AND INSPECTIONS 

The OIG’s Office of Evaluation and Inspections (OEI) conducts short-term management and 

program evaluations (called inspections) that focus on issues of concern to the Department, 

the Congress, and the public. The inspection reports provide findings and recommendations 

on the efficiency, vulnerability, and effectiveness of departmental programs. 

OEI’s Boston Regional Office prepared this report under the direction of Mark. R. Yessian, 

Ph. D., Regional Inspector General. Principal OEI staff included: 

Boston Region Headquarters 

Joyce M. Greenleaf, Project Leader Jennifer Antico, Program Specialist 

Theodore T. Wall, Progam Analyst 

For copies of this report, please call 617/565-1050. 
Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

Medicaid Managed Care: 

The Use of Surveys as a 

Beneficiary Protection Tool 

JUNE GIBBS BROWN 

Inspector General 

MAY 1997 

OEI-01-95-00280 
EXECUTIVE SUMMARY 

PURPOSE: 

To assess the usefulness of beneficiary surveys as a beneficiary protection tool for 

Medicaid managed care. 

BACKGROUND 

State Medicaid agencies are increasingly enrolling their beneficiaries in full-risk 

managed care plans. Beneficiary surveys are among the many tools available to them 

as a beneficiary protection. These surveys can be of value in two basic ways: 

To Gain Insights on Plan Performance: State Medicaid agencies rely on health plans to 

provide the full range of medical services for their enrolled populations. Overseeing 

the plans’ abilities to provide appropriate services is an important agency function. 

Surveys present an opportunity to give agencies information about plan performance 

from the perspectives of enrolled beneficiaries. 

To Help Benejkiaties Make Informed Enrollment Decisions: Beneficiaries that are 

informed about their enrollment decisions and the choices they have are likely to be 

better protected than those who are less informed about their choices. Survey results 

present an opportunity to help beneficiaries learn from the experiences of others in 

managed care, thereby helping them to make an informed enrollment choice. 

This inspection seeks a deeper understanding of the usefulness of beneficiary surveys 

in the two ways described above. It is based on interviews with the Medicaid agency 

leadership in 11 States, site visits to 4 of those States, and a survey of the managed 

care plans enrolling Medicaid beneficiaries in all 11 States. 

FINDINGS 

Surveys of Limited Use as Ben-y Protection 

To date, surveys provided little useful information about plan performance to 

Medicaid agencies. Agency leadership found their survey results largely confirming 

what they knew from other sources. They also found them to be of questionable value 

with regard to the technical quality of care and were reluctant to base corrective 

actions on survey data. 

Surveys have yet to provide beneficiaries with information to help them choose a plan. 

In one groundbreaking survey, Medicaid beneficiaries found the survey data to be 

“unwieldy.” In general, little is known about what information beneficiaries would like 

to have when faced with a choice of plans. 

i 
Agencies and Plans Face Basic Hurdles in Survqbzg Medicaid Bene&iaries 

The intermittent nature of Medicaid eligibility is such that only about half of the 

Medicaid beneficiaries eligible in one month will remain eligible 12 months later. Low 

response rates also create problems for agencies and plans. Two-thirds of the plans in 

our survey reported response rates of 20 percent or less in their mail surveys. These 

hurdles make such surveys expensive to conduct in a credible fashion. 

Strategic Survey Uses Emerging 

Some agencies had some success in using more narrowly focused surveys aimed at 

answering questions such as: Are enrolled beneficiaries getting the orientation 

information they need from the health plans ? What information do beneficiaries want 

available to choose a health plan ? Others have presented their survey data in such a 

way as to allow comparisons to average plan scores. 

Notwithstanding Limitations, Plans Find Surv~s Usw 

In fact, plans conduct surveys even where their contract with the Medicaid agency 

lacks a specific survey requirement. Their survey results have led to a variety of 

operational improvements and policy changes. 

7&e HCFA should either establish a work group or technical advisory group on Medicaid 

benejiciary surveys or add surveys to the agenda of an t&ting group. Either group should 

provide policy-level guidance on how to make cost-flective use of beneficiary survq.~s. 

This group would provide a forum for more targeted policy-level discussion about the 

best use of surveys. At a minimum, the group should address the following questions: 

b Given that resources are limited, what level of emphasis should Medicaid 

agencies give to using surveys for overseeing performance and providing 

information for beneficiary education and choice? 

b How can beneficiary surveys be used more effectively as a means for 

overseeing the performance of managed care plans? 

b How can surveys be used more effectively to provide information meaningful 

to beneficiaries faced with a choice of health plans? 

b How can the agencies’ contracts with plans be used to best benefit the 

agencies’ needs for survey information? 

b How can Medicaid agencies best deal with the hurdles inherent in surveying a 

Medicaid population? 

ii 
The HCFA should devote greater attention to how the Medicaid agencies are using 

ben@ciary survqs. It should revise its written guides for reviewing and monitoring 

Medicaid managed care init&ivtz~ to call attention to the knportance of using ben@cimy 

surveys in more focuse4&strategic ways. 

Toward that end, it should consider adding the questions below to its guides: 

b What does the agency aim to achieve with its beneficiary surveys? To what 

extent is it related to beneficiary protection? 

b How narrowly has it defined the survey’s purpose? Could a more targeted 

use of a survey enhance its value as a beneficiary protection tool? 

b Will the survey fill a gap in the agency’s system for overseeing quality in the 

health plans? Will it provide information that the agency knows beneficiaries 

rely upon in choosing a health plan? 

b Does the agency have a cost-effective plan for overcoming the hurdles 

associated with surveying Medicaid beneficiaries? 

b Has the agency weighed the relative costs of surveying beneficiaries with the 

relative usefulness of the survey results? 

CoMMENTsoNmDm REPORT 

We solicited and received comments on the draft report from HCFA and the 

American Association of Health Plans (AAHP). We include the complete text of their 

comments in appendix C. Below we summarize the major comments on our 

recommendations and, in italics, summarize our responses. 

The HCFA partially concurred with our recommendation on establishing a group for 

policy-level guidance on making cost-effective use of beneficiary surveys. It noted its 

existing Medicaid Managed Care Technical Advisory Group could fulfill this role. It 

also noted its ongoing collaboration with the Agency for Health Care Policy and 

Research, which is leading the Consumer Assessment of Health Plans Study. The 

AAHP supported our recommendation and suggested HCFA include some health plan 

representatives in the group. We recognize that HCFA’s existing group could be 

appropriate to provide a forum for policy-level discussion about the use of surveys. We 

appreciate AAHP’s support of our recommendation and urge HCFA to consider including 

health plan representatives in its group for this discussion. 

Our second recommendation called for HCFA to devote greater attention to how 

Medicaid agencies are using surveys and to revise its written guides toward that end. 

The HCFA concurred with points made in our text: that some surveys have multiple 

objectives and need to be more focused. The HCFA plans to include a special session 

on surveys in its annual Medicaid Managed Care College and will stress the 

... 

111 
importance of surveys in its technical assistance to HCFA regional office and State 

Medicaid staff. The AAHP supported our recommendation. We welcome HCFA’.s 

plans for survey sessions at its Managed Care College and through its ongoing technical 

assistance. We urge it to use these venues to stress more strategic and focused uses of 

surveys. And we reemphasize the need for HCFA to revise its written guides for reviewing 

Medicaid managed care initiatives. In its comments, HCFA did not indicate that it would 

revise these guides along the lines we suggested. We appreciate AAHP’s support of our 

recommendation. 

iv 
TABLE OF CONTENTS 

EXECUTIVESUMMARY 

INTRODUCJTION ................................................. 1 

FINDINGS ...................................................... 3 

� Surveys of Limited Use as a Beneficiary Protection ...................... 3 

� Basic Hurdles Faced in Surveying Medicaid Population ................... 6 

� Strategic Survey Uses Emerging .................................... 8 

� Notwithstanding Limitations, Plans Find Surveys Useful .................. 9 

RECOMMENDATIONS ........................................... 11 

COMMENTS ON THE DRAFT REPORT ............................. 14 

APPENDICES 

A: Background on Medicaid Managed Care ............................ A-l 

B: Methodology ................................................ B-l 

C: Complete Comments on the Draft Report ......................... C-l 

D: Endnotes ................................................... D-l 
INTRODUCTION 

PURPOSE 

To assess the usefulness of beneficiary surveys as a beneficiary protection tool for 

Medicaid managed care. 

Surveys of consumers enrolled in managed care plans are becoming increasingly 

popular. They can provide important insights to the managed care experience. The 

surveys’ results can be used to market a specific health plan, promote managed care 

as a delivery system, or simply answer research questions. In fact, the Health Care 

Financing Administration (HCFA) often requires Medicaid agencies implementing ’ 

managed care waivers to conduct surveys. And for any of the agencies that are 

increasingly enrolling their beneficiaries into managed care plans (see appendix A for 

more background on Medicaid and managed care), surveys can be helpful in another 

way: beneficiary protection. In that regard they can be of value in two basic ways: to 

gain insights on plan performance and to help beneficiaries make educated enrollment 

choices. 

To Gain Insights on Plan Pe$onnance: State Medicaid agencies rely on health 

plans to provide the full range of medical services for their enrolled 

populations. Overseeing the plans’ abilities to provide appropriate services is 

an important agency function. Surveys present an opportunity to give agencies 

information about plan performance from the perspectives of enrolled 

beneficiaries. 

To Help Benejiciaties Make Informed Enrollment Deckions: Beneficiaries that 

are informed about their enrollment decisions and the choices they have are 

likely to be better protected than those who are less informed about their 

choices. Survey results present an opportunity to help beneficiaries learn from 

the experiences of others in managed care, thereby helping them to make an 

informed enrollment choice. 

Of course, surveys are not the only tools available to State agencies for 

beneficiary protection; they are one of many. ’ For example, Medicaid agencies rely 

on their contracts, plan site visits, complaints, focus groups, and encounter data, 

among others means, to protect beneficiaries. But surveys, like complaints, are one of 

the few tools that actually require some direct interactions between the beneficiaries 

and the agency. 

1 
THIS INQUIRY 

The HCFA recently issued a guide for States to help them develop customer 

satisfaction surveys.’ It provides some tips on developing surveys, an inventory of 

recent surveys, and contact people at the agencies for those interested in further 

information. 

This inspection seeks a deeper understanding of the usefulness of surveys as 

beneficiary protection tools in the two ways described above: plan performance and 

beneficiary education. Such an understanding is becoming increasingly important as 

the proportion of Medicaid beneficiaries enrolled in managed care plans continues to 

mount. Our focus in this inspection is on mail and telephone surveys rather than 

focus group surveys. We know little now about how effective such mail and telephone 

surveys have been as a beneficiary protection tool. This inspection also offers some 

assessment, from the perspective of the Medicaid managed care leadership, of the 

surveys included in HCFA’s recent guide. 

In conducting this inspection, we interviewed the Medicaid agency leadership in 

11 States that have enrolled significant numbers of Medicaid beneficiaries into full-risk 

managed care plans. 3 All of the 11 States in our study have had experience in 

surveying, directly or through contractors, their Medicaid populations. Most often, 

they conducted satisfaction and disenrollment surveys and used a mail or telephone 

approach. A few agencies also conducted some focus groups. For this inspection, we 

drew on the 11 agencies’ experiences with mail and telephone surveys rather than 

focus groups. We also conducted our own survey of full-risk managed care plans 

enrolling Medicaid beneficiaries in those States. (See appendix B for more detailed 

information on our methodology). 

We conducted this inspection in accordance with the Quality Standards for 

Inspections issued by the President’s Council on Integrity and Efficiency. 

2 
FINDINGS 

Surveys, as current& used by the 11 State Medicaid agencies in our sample, are of limited 

value as a beqticiq protection. 

b Surveys provide little useful information about plan performance to Medicaid 

agencies. 

All of the 11 agencies in our study 

had experience with surveying beneficiaries 

over the past 2 to 3 years. Eight of the 

11 conducted recent surveys that included 

goals of assessing plans’ performance; 3 of 

these surveys were still underway at this 

writing. The bulk of their experience was 

with satisfaction and disenrollment surveys. 

According to the agency leadership, 

neither satisfaction nor disenrollment 

surveys provided much in the way of new 

insights into plan performance. The 

insights they gained about plan 

performance from satisfaction surveys were 

largely limited to enrollee perceptions 

about customer service issues, such as 

waiting times. While relevant to a plan’s 

performance, the agency officials pointed 

out that they were already aware of such 

concerns through other means, such as 

complaints and contract compliance visits. 

Disenrollment surveys were similarly 

disappointing. Although thousands of 

beneficiaries disenrolled from plans every 

month, only a small percent--often 

1 percent or less--disenrolled voluntarily; 

most disenrolled due to a loss of eligibility. 

So the pool of potential survey - -

The 1995 Minnesota HDI Sumy 

in the spring of lPPS, the Minnesota 

Health Data Pnstituie (HDI)--a 

public-private organization created by 

the State in 1993 to improve health 

care for Minnesotans--surveyed 

17,591 Minnesotans enrolled in 46 

plans, including Medicaid, Medicare, 

arid the private& insured. HDl’s 

goals were to provide “‘standardized 

measurement of health plaan 

performance” to consumers, 

purchasers, and plans, 

After the survey, HDI contracted for 

the proje& evahation. One issue 

the evaluation explored was whether 

the survey results helped policymakers 

monitor and evaluate health system 

perJonnance+ In the evaluation 

report, policymakers did not identify 

any ways the survey helped them 

monitor pefownance. 

(Source: Mimesotu Health Data &.&A& 1995 

Corz.wner Sufvty Evaluation Report, Ju+ 1996) 

respondents from which to gain insight into the performance of any one plan was 

limited. One agency used over 100 different codes to reflect reasons for 

disenrollment. With so many codes for so few disenrollees, it is unlikely a pattern 

would emerge to provide insight into plan performance. 

Agency leadership also indicated that surveys had little value with regard to 

technical quality of care (meaning examinations, follow-through and treatment).4 For 

assessing technical quality of care, they relied more heavily on other indicators of 

performance, such as encounter data, medical record reviews, contract compliance, 

3 
and complaints. ’ Surveys are a limited tool to gain insight into so complex a topic as 

the quality of care. Agency leadership questioned the meaning of satisfaction data: 

could care that is truly managed lead to lower satisfaction due to restricted access to 

specialists? They questioned the link between satisfaction and quality. They noted 

that surveys were not good sources for identifying poor quality-of-care. Indeed, high 

satisfaction and poor quality can occur simultaneously, as the Massachusetts agency 

learned when its plans with the highest scores on quality indicators scored lower in 

satisfaction.6 Two officials noted that while the plans’ performance matters to the 

agency, it is the physicians that matter most to the beneficiaries. 

What surveys revealed about the plans versus what they revealed about the 

physicians or group practices complicated their usefulness in certain markets. For 

example, in the Twin Cities area of Minnesota and certain markets in Missouri, plan 

networks overlap considerably. This means many of the same physicians and group 

practices participate in the same plans, so plan-specific interpretations of survey data 

become questionable. 

To date, the agencies have been unable to rely on survey data to make many 

useful distinctions over time or among plans. For example, the 11 States in our study 

have yet to conduct the same survey twice, which would allow them to compare results 

over time.7 Also, survey results, such as satisfaction scores, tended to cluster at the 

high end. Officials from two agencies’ leadership noted that their surveys would 

establish a baseline for future comparisons--in one case a previous survey aimed to do 

the same but low response rates rendered it unusable. 

In most cases, agency officials reported they would be reluctant to direct 

corrective actions to plans based on survey data alone. One agency official referred to 

the survey data as “too anecdotal” to base any formal corrective actions. An official in 

another State intended to establish minimum performance levels, or benchmarks, for 

future surveys. The agency would penalize or direct some action against plans failing 

to score above the benchmarks. For satisfaction--on which all the plans in the State 

had scored in the 90s on the most recent survey--the official was considering a 

benchmark of just 70 percent, thus it is unlikely the benchmarks would be unmet. 

Most agencies did, however, share their survey results with their plans and expected 

the data to prompt improvements. 

4 
b The surveys have yet to provide beneficiaries with information to help them 

choose a plan. 

Of the 11 Agencies in our study, 

6 had completed or were in the process 7he 1995 A4hwmta HDI Survey 

of completing surveys that included 

getting information back to beneficiaries In October of 1995j 60 newspapers 

as a goal. At the time of this writing, all across the State published the results of 

but one of these surveys were still the survey as an insert to the Sunday 

underway. edition. The results were also available 

in libran-es. 

In Minnesota, county staff 

educate the Medicaid beneficiaries about 

their health plan choices. An evaluation of the HDI survey found that while widely 

disseminated, consumers in general did not find it especially helpful in choosing a 

health plan.* It also found, through a focus group, that Medicaid beneficiaries found 

the information to be “unwieldy,” a sentiment reflected in the two counties we visited. 

The staff there responsible for counseling beneficiaries about their plan choices did 

not use the HDI survey results to help beneficiaries make a choice, saying it contained 

“too much information to be useful.” 

Though widely heralded as a groundbreaking survey, the HDI experience raises 

some important questions about what is really known about what consumers, and 

particularly Medicaid beneficiaries, want to know when faced with a choice of plans. 

How important is survey data, such as satisfaction scores or waiting times? The 

Massachusetts agency has a project underway that aims to measure how useful its 

report cards are to beneficiaries faced with a choice of plans. The report card 

includes data from satisfaction surveys, as well as some quality indicators and other 

information. That agency plans to survey its newly enrolled beneficiaries and assess 

the report card’s impact on their choice of plans. In Oregon, where a consumer 

scorecard project funded by the Federal government has been underway, officials 

learned that consumers are “primarily interested in the quality of services offered by 

their doctors and hospitals, and less interested in data on their health plan itself.“’ 

Other researchers have come to different conclusions: “...research by the Picker 

Institute and others seems to show that patients and consumers don’t yet know what 

measures are actually meaningful to them.“” Some research has suggested that 

consumers tend to find satisfaction data suspect and prefer relying on their own family 

members and friends when choosing a plan.ll 

5 
Both agencies and plans face basic hurdles in survqirzg the Medicaid population. 

w Intermittent Eligibility 

Historically, Medicaid eligibility 

has been tied to programs that provide 

temporary assistance such as Aid to 

Families with Dependent Children. 

Thus, beneficiaries tend to come in and 

out of eligibility as their income and 

family composition changes. The 

HEDIS Work Group compiled data on 

the extent to which beneficiaries were 

continuously enrolled over a 36-month 

period (see table l).l’ That data shows 

a steady decline in continuous 

enrollment, with just 55 percent enrolled 

for 1 year, 29 percent for 2 years, and 

17 percent for 3 years. 

The relatively short lengths of 

time that beneficiaries are enrolled limits 

the extent of their opportunity to have 

much contact with or receive care from 

the plans. Thus the universe from which 

b 

Percent 

Month Continuously 

Enrolled 

1 100% 

6 14% 

12 55% 

24 29% 

36 17% 

Note: Based on 8 States’ experiences 

from 1O/91 through 12193 

Source: Medicaid HEDIS DraJ?, July 

199.5 

an agency or plan would draw its survey sample is also limited. In fact, 78 percent of 

plans responding to our survey identified intermittent eligibility as an obstacle to using 

surveys. Thirty-eight percent identified it as a major obstacle. The plans identified 

intermittent eligibility as an obstacle more often than anything else. In the words of 

some plan officials, members are simply “not enrolled long enough to use services” and 

“on and off again eligibility...do not lend themselves to a meaningful managed care 

experience.” 

b Low Response Rates 

Both plans and agencies reported low response rates with their surveys. In fact, 

according to our survey of plans, 66 percent of the plans that had conducted a mail 

survey of their Medicaid enrollees had response rates of 20 percent or less. Low 

response rates were common in the agencies, too. Indeed, in Massachusetts the 

agency was able to contact just 7 percent of its sample in one survey. The Missouri 

agency expanded its sample after the response rate to its first mailing was just 

8 percent. And early experiences with surveys in Florida, Oregon, and Utah had 

response rates too low for the survey results to be usable. 

Many factors contribute to low response rates within the Medicaid population. 

Medicaid populations tend to be more transient than other populations, meaning 

6 
agencies and plans often contend with outdated addresses and telephone numbers. In 

Ohio, the agency found about 70 percent of its sample population to be either invalid 

or with incorrect phone numbers. Differences in language and literacy skills--either on 

the part of the agency or plan administering the survey or on the part of the 

beneficiary responding to the survey--also lead to low response rates. Two-thirds of 

the plans responding to our survey identified literacy problems and half identified 

language problems as obstacles to using surveys. 

b costs 

Conducting credible surveys is costly. It requires well-thought out methodology, 

and sampling designs, carefully constructed questions, adequate response rates, and 

careful analysis. Each of those stages can tax a plan or agency’s staff, time, and funds. 

For example, agencies and plans often lack the experience or skills needed to develop 

a credible questionnaire.13 One agency official pointed out that its earlier experience 

with surveying enrollees failed in large part due to poor survey design. Thirty-two 

percent of the plans in our survey identified the lack of a suitable off-the-shelf 

questionnaire as an obstacle to using surveys; 23 percent identified their limited 

capacity to develop their own questionnaires as an obstacle. 

Even if an agency or plan has a suitable questionnaire, actually carrying out the 

survey can be labor-intensive. It can involve making calls, stuffing and labeling 

envelopes, tracking responses, targeting follow-up, and analyzing responses, among 

other tasks--or at a minimum: overseeing the contractor’s work. One agency shared 

the costs of its mail survey by requiring the plans to mail the questionnaire themselves. 

Forty percent of the responding plans in our survey identified limited capacity to 

administer surveys and 33 percent identified limited capacity to assess results as 

obstacles to using them. 

Among agencies, costs are also high. For example, to achieve respectable 

response rates, the Oregon agency used an intensive 5-step process that involved, in 

multiple languages: an advance letter mailing, an initial survey mailing, a reminder 

postcard, a second survey mailing, and a second reminder postcard or phone call. The 

Oregon agency’s intensive effort paid off with a 63 percent response rate. Officials 

there note, however, that “it is too much to do annually.” 

Other agencies have undertaken similarly intensive survey projects. For 

example, the Utah agency also undertook a survey that involved agency and field staff­

-as opposed to a contractor--making calls to a sample of 1,000 enrolled beneficiaries. 

They completed 579 surveys over a 2-month period, resulting in an adjusted response 

rate of 73 percent.r4 The Florida agency is also in the midst of its intensive survey 

effort that involves multiple mailings, similar to the Oregon effort. As of this writing, 

Florida has a response rate in the low 30s and is extending its surveying by 5 weeks. 

7 
Some agencies are beginning to use surveys in strategic ways, with potentially promising 

results. 

b More Focused Surveys 

Much of agencies’ experiences to 

date has been with large-scale 

satisfaction surveys that aimed to meet 

multiple objectives. But some agencies 

have conducted surveys with a narrower 

focus and found the results to be quite 

useful. In the words of one agency 

leader, “The usefulness of surveys is 

likely to be closely correlated to how 

focused they are.” 

The survey described at the right 

is an example of a focused survey 

designed to meet specific information 

needs of the agency. We also learned of 

another focused survey in Utah, where 

agency staff spent some late nights in 

hospital emergency rooms interviewing Medicaid beneficiaries. That survey gave the 

agency new insights into inappropriate emergency room usage and led to changes in 

its policies on emergency room copayments and reimbursement. 

Other agencies have had similar success with narrowly focused surveys. For 

example, the Rhode Island agency conducted two surveys: one to determine what 

factors beneficiaries relied upon to select their health plans and one to determine 

what factors they relied upon to choose a primary care physician. The Massachusetts 

enrollment broker reduced its field staff from 45 to 10 based in part on information 

learned through a focused survey. 

b Better Presentation of Survey Data 

As we have seen, to date the agencies in our study have not repeated the same 

survey, thereby precluding them from presenting survey results over time. 

Nonetheless, some agencies have presented their survey data in such a way as to 

maximize its utility. For example, the Oregon agency divided its presentation of 

survey results between plans in urban and rural areas. It included the average scores 

for plans in each area, and used colors to indicate which plan’s scores were above or 

below average. Of course, whether the average score was good or bad is unknown. 

Nevertheless, presenting the data with an average provided some context in 

understanding one plan’s performance relative to the average. 

8 
Notwithstanding the limitations of benejkiiuy surveys, health plans still jind them to be of 

some use in identijjhg and responding to enrollee concerns. 

b Plans Conduct Surveys Even Without Contractual Requirements 

Eight of the 11 agencies in our sample required that the plans they contract 

with conduct surveys. According to our survey, the majority of plans responding from 

all 11 States had surveyed Medicaid beneficiaries in the past 2 years.r5 Yet even 

plans without the contractual requirement 

responding plans (85 percent) from those 

average, the plans responding from those 

past 2 years. Overall, 83 percent of the 

including Medicaid, an average of about 

The actual contract requirements 

regarding surveys varied from agency to 

agency, but overall, the contract 

language regarding surveys was quite 

limited. Seven of the eight contracts 

required the plans to conduct the 

surveys annually and to report results to 

the agencies, but none specified 

methodologies for the plans to use. Two 

contracts specified that the survey must 

cover “perceived problems in quality, 

availability, and accessibility” of health 

care services, but the others lacked 

specific requirements for the surveys’ 

contents. 

The lack of many specific 

contractual requirements allows plans 

great latitude in surveying beneficiaries, 

and we found great variation among the 

conducted surveys. In fact, the majority of 

3 States conducted surveys anyway. On 

States conducted about 4 surveys over the 

responding plans surveyed their memberships, 

5 times over the past 2 years. 

Institutionalizing the Sw~ty &3txxs 

One Medicaid-only plan we visited had 

institutionalized its surveys so that each 

quarteq it sent out satisfaction surveys 

to a 25 percmt random sampEe of 

members who had been continuously 

enroiled for tj months. Each month it 

sent out disenrolbnent surveys to 

fOUpercent of voluntary disenrollees, 

Both surveys are in English and 

Spanish, Response rates hovered 

around 10 percent for these surveys, 

and the plan considered the results a 

roiling mme of feedback to share with 

itsproviders. 

plans’ approaches to surveying.16 For 

example, for their most recent survey, 64 percent of plans conducted a mail survey, 

24 percent telephone, and about 4 percent each self-administered, in-person, and focus 

group. Plans also often surveyed their whole membership, rather than just Medicaid 

beneficiaries. In fact, 44 percent of responding plans reported that their most recent 

survey included non-Medicaid enrollees. Targeting certain Medicaid enrollees for 

recent surveys was also common among responding plans. Of the 80 percent that 

targeted in some way, 51 percent targeted Medicaid disenrollees; 45 percent, Medicaid 

enrollees in certain geographic areas; 44 percent, users of certain Medicaid services; 

33 percent, users of certain physicians; and 26 percent, Medicaid enrollees with certain 

diagnoses. 

9 
w Plan Surveys Lead to Improvements 

Even though plans face the same 

hurdles--such as low response rates--as 

agencies in surveying their Medicaid 

enrollees, the plans in our own survey 

routinely surveyed their memberships. 

Their surveys gave them some cues 

about what mattered to their 

memberships, thereby guiding their 

improvements and helping them 

compete.r7 

The operational focus of plans’ 

surveys is evident by the variety of 

changes they prompted within the plans. 

Indeed, 56 percent of the responding 

plans explained their surveys’ usefulness 

in terms of the improvements to which 

they led. This reflects national trends in 

health plans using surveys: in 1995, 

virtually all plans--99 percent--reported 

using of surveys for quality improvement, 

provider feedback, and marketing.‘* 

According to our survey, plans 

made both operational and policy 

changes based on insights they gained 

from surveying their Medicaid enrollees. 

For example, they expanded their 

provider networks, translated written 

materials, revised prescription drug 

policies, added toll-free telephone lines, 

changed referral policies, and increased hours 

Percent of Plans Rating Surveys as 

Us&i for IdentifLig Medicaid Emlke 

Con~ems About.-

Quality of Care 75% 

(interpersonul communication, listening 

responsiveness, manne;r, courtesy) 

Quality of Customer Service 

73% 

(responsiveness to questiovts, concerns, 

and complaints) 

Waiting Times 6.5% 

(for appointments, and in waiting and 

exam rooms) 

Access 55% 

(to specialty care such as mental 

health, prenatal, substance abuse, and 

to emergency rooms) 

Technical Quality ofCare 365% 

(examinations>follow-through, and 

treatment) 

Note: Categories a&pted$om the oraft Medicaid HEDIS 

document, .iu& 1995. N=69-79. 

Source: 1996 DIG Sutvq. 

of operation, among others. A few 

plans also noted that they used surveys to give feedback to their physicians, and in 

some cases, had established incentives for physicians who scored high on surveys. 

10 
RECOMMENDATIONS 

This report raises some questions about the usefulness and cost-effectiveness of 

surveys in protecting beneficiaries from the vulnerabilities associated with full-risk 

managed care plans. There are two basic ways in which surveys can serve as a 

beneficiary protection tool: (1) to provide Medicaid agencies with information on the 

performance of the plans enrolling Medicaid beneficiaries, and (2) to provide the 

beneficiaries facing a choice of health plans information to help them choose. Given 

the partnership nature of the Medicaid program, the Federal and State governments 

should work cooperatively to identify ways in which surveys can best be used as a costeffective beneficiary protection tool. Toward that end, we direct the following 

recommendations to the Health Care Financing Administration: 

The HCFA should either establkh a work group or technical advisory group on Medicaid 

benejkiary survqs or add surveys to the agenda of an existing group. Either group should 

provide policy-level guidance on how to make cost-effective use of benejkia~ surveys. 

Such a group would provide a forum for more targeted policy-level discussion 

amongst State Medicaid leadership on how to best use surveys as a beneficiary 

protection. Medicaid policymakers need to figure out how surveys can best fit in with 

the array of beneficiary protection tools, such as encounter data and HEDIS and 

contracts, at their disposal. The involvement of officials at the policy level is key, 

because without direction from that level, the danger exists that the group will become 

consumed with the methodological minutiae of conducting surveys rather than the 

information needs they can fill. 

The survey group should aim to build on the existing “Customer Satisfaction 

Surveys for Medicaid Beneficiaries: A State Guide” recently issued by HCFA. At a 

minimum, the group should address the following questions with an eye toward 

maximizing surveys’ potential as a beneficiary protection tool: 

b Given that resources are limited, what level of emphasis should Medicaid 

agencies give to using surveys for overseeing performance and providing 

information for beneficiary education and choice? 

Medicaid agencies need to maximize the potential of surveys as a 

beneficiary protection tool, and this will involve tough choices. Using a 

single survey to serve multiple objectives can limit its effectiveness in 

meeting any one objective. Conducting a survey and having credible 

data takes an intensive effort. Medicaid agencies need to consider the 

strengths and weaknesses of their existing beneficiary protection systems 

and weigh the relative merit of using a survey for oversight or 

educational purposes. 

11 
b How can beneficiary surveys be used more effectively as a means for 

overseeing the performance of health plans? 

Medicaid agencies rely on a variety of data and information to assess the 

performance of managed care plans. Surveys could serve to provide 

information about topics which other oversight tools fail to address or 

address insufficiently. For example, surveys could be used to gain a 

better understanding of the experiences of the chronically ill or of those 

enrolled in managed care but not receiving services (do they consider 

themselves healthy? do they know they are enrolled? do they know how 

to access services?). 

b How can surveys be used more effectively to provide information meaningful 

to beneficiaries faced with a choice of health plans? 

The Agency for Health Care Policy and Research (AHCPR) is engaged 

in a 5-year Consumer Assessment of Health Plan Survey project aimed 

at using surveys to develop report cards for consumers facing a choice of 

plans. l9 What can Medicaid agencies learn from this project? What is 

known about what types of information Medicaid beneficiaries need 

when making a choice? 

b How can the agencies’ contracts with managed care plans best benefit the 

agencies’ needs for survey information? 

Medicaid agencies have an opportunity with their contracts to call for 

the plans to conduct specific surveys and even to stipulate the sampling 

design, the question phrasing, and the survey methodology. How shoulc 

agencies best take advantage of that opportunity? Should they aim to 

have all their plans conduct uniform surveys according to contract 

specifications? 

b How can Medicaid agencies best deal with the hurdles inherent in surveying a 

Medicaid population? 

Conducting a credible survey always requires careful consideration to a 

variety of methodological issues, such as sampling and approach 

(telephone, mail, or in-person, for example). But surveying a Medicaid 

population presents unique hurdles, such as short and discontinuous 

eligibility. 

12 
The HCFA should devote greater attention to how the Medicaid agencim are using 

bene&imy survqs. It should revise its written guides for reviewing and nwnitoring 

Medicaid managed care initiativ~ to call attention to the importance of using benejicky 

surveys in more focused, strategic ways. 

The HCFA already identifies surveys and their uses as a point of review in its 

guides for reviewing and monitoring Medicaid managed care initiatives.20 It could, 

however, enhance the attention it gives to surveys by incorporating greater discussion, 

inquiry, and assessment of agencies’ use of surveys at each point for formal review or 

monitoring. Toward that end, it should consider adding the questions below to its 

guides: 

b What does the agency aim to achieve with its beneficiary surveys? To what 

extent is it related to beneficiary protection? 

b How narrowly has it defined the survey’s purpose? Could a more targeted 

use of a survey enhance its value as a beneficiary protection tool? 

b Will the survey fill a gap in the agency’s system for overseeing quality in the 

health plans? Will it provide information that the agency knows beneficiaries 

rely upon in choosing a health plan? 

b Does the agency have a cost-effective plan for overcoming the hurdles 

associated with surveying Medicaid beneficiaries? 

b Has the agency weighed the relative costs of surveying beneficiaries with the 

relative usefulness of the survey results? 

13 
COMMENTS ON THE DRAFT REPORT 

We solicited and received comments on the draft report from the Health Care 

Financing Administration (HCFA) and the American Association of Health Plans 

(AAHP). We include the complete text of the detailed comments in appendix C. 

Below, we summarize HCFA’s and AAHP’s major comments on the recommendations 

and then, in italics, offer our responses. 

RECOMMENDATIONSFROMTHE DRAFT REPORT 

T%eHCFA should either establish a work group or technical adtiory group on Medicaid 

benej?ciary surveys or add surveys to the agenda of an existing group. Either group should 

provide policy-level guidance on how to make cost-flective uses of benejZiary surveys. 

The HCFA partially concurred with this recommendation. It pointed out that 

its existing Medicaid Managed Care Technical Advisory Group has a work group 

currently working on consumer information and surveys. The HCFA agreed with our 

assessment that agencies often conduct surveys for multiple purposes but disagreed 

with our assessment that these were often of limited value. The HCFA pointed out 

that it is not feasible for States to develop single-purpose surveys. It further noted 

that it believes States have obtained some meaningful information from surveys, even 

where that information was not what the State intended to acquire. And it noted its 

ongoing collaboration with the Agency for Health Care Policy and Research, which is 

leading the Consumer Assessment of Health Plans Study. 

The AAHP supported this recommendation and suggested that HCFA include 

some health plans in this work group. It reinforced the importance of weighing the 

relative costs of surveys with the relative usefulness of the survey results for not only 

State agencies but also health plans. 

We recognize that HCFA’s Technical Advisory Group could be an appropriate 

group to provide policy-level guidance on the cost-effective use of beneficiary surveys and 

we urge it to continue collaborating on the Consumer Assessment project. We maintain, 

however, that based on this inquiry, narrowly focused surveys have been more useful to 

State agencies as a beneficiary protection tool. 

We appreciate AAHP’s support for our recommendation and urge HCFA to 

consider including health plan representatives in its Technical Advisory Group to discuss 

the cost-effective use of surveys. 

14 
The HCFA should devote greater attention to how the Medicaid agencies are using 

benejkiay surveys. It should revire its written guides for reviewing and monitoring 

Medicaid managed care intitivm to call attention to the importance of using benejitiry 

surveys in more focus4 strategic ways. 

The HCFA concurred with points raised in our findings and text: that some 

surveys have multiple objectives and need to be more focused. It pointed out that 

surveys are a very important part of State agencies’ requirements under the Quality 

Assurance Reform Initiative, which is being replaced with a system of standards for 

both Medicare and Medicaid. It anticipates that surveys will be an important 

component of that new system. The HCFA plans to include a special session on 

survey development and use of survey data in its annual Managed Care College and 

will stress the importance of surveys in its technical assistance to HCFA regional office 

and State Medicaid staff. 

The AAHP supported this recommendation. 

We welcome HCFA ‘s plans for survey sessions at its Managed Care College and 

through its ongoing technical assistance and urge it to use these venues to stress more 

focused and strategic uses of surveys. And we reemphasize the need for HCFA to revise 

its written guides for reviewing and monitoring Medicaid managed care initiatives. In its 

comments, HCFA did not indicate that it would revise these guides along the lines we 

suggested. 

We appreciate AAHP’s support of our recommendation. 

15 
APPENDIX A 

BACKGROUNDONMEDICAIDMANAGEDCARE 

Medicaid Expansion into Managed Care Programs 

Over the past 15 years, States have increasingly used managed care to provide 

medical services for Medicaid beneficiaries. This trend has accelerated in the past few 

years: in 1991, 2.7 million Medicaid beneficiaries were enrolled in managed care, by 

1993, that number grew to 4.8 million, and in 1996, 13 million. As of June 1996, 

39 percent of all Medicaid beneficiaries were enrolled in some kind of managed care 

arrangement. 

To date, States have primarily enrolled adults and children in low-income 

families into managed care, whereas aged or disabled beneficiaries remain under feefor-service systems. By 1996, over 500 managed care organizations were providing 

services to 13 million Medicaid beneficiaries. 

The movement to enroll Medicaid beneficiaries in managed care began in 

earnest in the early 198Os, as States experienced significant fiscal pressures due to 

rising Medicaid costs. While States viewed managed care as a way to contain 

Medicaid costs, they were constrained by Federal standards required for Medicaid 

enrollment in managed care. 

In response to mounting concerns, Congress allowed States greater flexibility to 

deviate from those standards through amendments to the 1981 Omnibus Budget 

Reconciliation Act.21 For example, the amendments allow States to pursue freedomof-choice waivers (under section 1915 of the Social Security Act) that release them 

from certain Federal provisions, such as the free-choice-of-provider provision. To 

date, 42 States have freedom-of-choice waivers. 

States also can receive research and demonstration waivers under section 

1115 of the Social Security Act. Since 1992, many States have aggressively pursued 

such waivers. States implementing or pursuing 1115 waivers often extend, as a part of 

their demonstration, insurance benefits to those not otherwise eligible for Medicaid, 

such as the working poor and their families. As of this writing, HCFA has approved 

18 research and demonstration waivers. Of those, 12 States have implemented their 

programs, 5 are pending implementation, and one has no plans to implement.22 The 

HCFA is currently reviewing nine States’ applications.23 

A -1 
Defining Managed Care 

Although managed care organizations vary, they generally feature a focus on 

primary, preventive health care and care coordination. That focus is believed to 

improve care and access for enrollees. It is also thought to promote cost containment, 

thus slowing the rate of increase in health care spending. 

The managed care organizations enrolling Medicaid beneficiaries can generally 

be defined as fitting into one of two basic types: health maintenance organizations 

and fee-for-service primary care case management (PCCM) programs. Both types 

feature coordinated care. But each carries a different level of financial risk. Health 

maintenance organizations (hereafter referred to as health plans or plans) are full-risk 

plans that contract with Medicaid for a fixed fee per person and provide 

comprehensive services. 24 PCCM programs comprise providers, usually primary care 

physicians, willing to serve as gatekeepers and take responsibility for approving and 

coordinating enrollees’ care. Medicaid pays PCCM providers on a fee-for-service 

basis, but they receive a case management fee to cover their added responsibilities. 

Thus PCCM providers are at no financial risk. 

Some agencies also contract with plans separately for certain aspects of care, 

such as mental health, substance abuse, and dental care. The arrangements for these 

“carve-outs” vary in terms of the services included and the level of financial risk the 

plan assumes. 

A -2 
APPENDIX B 

We drew on three major sources of data for this study: (1) interviews with the 

Medicaid managed care leadership in 11 States, (2) a survey of the full-risk managed 

care plans enrolling beneficiaries in those 11 States, and (3) the model contracts used 

by the Medicaid agencies in contracting with the managed care plans. 

II Participating State Medicaid Agencies 

At the outset of our study, we chose the 13 State Medicaid programs that had 

made a significant commitment to enrolling their beneficiaries in full-risk managed 

care plans, represented different geographic areas of the country, and with waivers in 

various stages of approval or implementation. Medicaid agencies in 11 of these States 

participated: Arizona, California, Florida, Massachusetts, Minnesota, Missouri, New 

York, Ohio, Oregon, Rhode Island, and Utah. Two State agencies declined to 

participate. Of the participating 11 States, 6 had approved and implemented 

1115 waivers (Arizona, California, Minnesota, Ohio, Oregon, and Rhode Island), 2 had 

approved waivers pending implementation (Florida and Massachusetts), and 3 had 

pending waivers (Missouri, New York, and Utah). Together, the 11 States represent 

about 37 percent of all Medicaid beneficiaries in the country. The results of our study 

are not generalizable to other State Medicaid agencies. 

Interviews and Site Viits 

We conducted structured telephone interviews with the Medicaid managed care 

leadership in eight of the State Medicaid agencies. We visited the Medicaid agencies 

in five of the States on one or more occasions to conduct the interviews in person and 

meet with additional agency staff, advocates, and representatives of the State’s 

legislature and managed care plans. We conducted site visits in California, 

Massachusetts, Minnesota, Rhode Island, and Utah. During both the telephone 

interviews and site visits we collected any relevant documents such as survey reports, 

contracts, and evaluations. 

Contract Review 

We reviewed the model contracts in use at the time of our interviews for 

requirements related to surveys. 

Survey ofFull-Risk Managed Care Plans in 11 States 

We received lists of full-risk managed care plans enrolling Medicaid 

beneficiaries from each of the 11 agencies in our sample. We then sent, via facsimile, 

a questionnaire asking about the plan’s recent survey activity. We sent a second 

survey to nonresponding plans after about 3 weeks, and then followed up by telephone 

B-l 
with remaining nonresponding plans after about 6-8 weeks. The universe for this 

survey included 162 plans in 11 States. Eight of the 162 plans were invalid because 

they no longer enrolled Medicaid beneficiaries, therefore our adjusted universe 

included 154 plans. We received 105 completed questionnaires in time to be included 

in our analysis. Our adjusted response rate was 68 percent. 

We attempted to conduct a non-respondent analysis to determine how 

nonresponding plans differ from responding plans, if they do at all. Because we 

lacked certain data elements, such as tax status and size of enrolled population for 

some plans, we were unable to complete the analysis. The results of our health plan 

survey are not generalizable to all plans. 

B-2 
APPENDIX C 

Com4~m-s 0Nm DRAFT REPORT 

In this appendix, we present in full the comments from the Health Care 

Financing Administration and the American Association of Health Plans. 

C-l 
DEPARTMENT OF HEALTH &HUMAN SERVICES Health Care Financing Administration 

The Administrator 

Washington, D.C. 20201 

DATE: APR 23 1997 

TO: June Gibbs Brown 

Inspector General 

FROM: Bruce C. Vlade 

Administrator 

SUBJECT: Office of Inspector General (OIG) Draft Report: “Medicaid Managed Care: 

The Use of Surveys as a Beneficiary Protection Tool,” (OEI-01-95-00280) 

. 

We reviewed the above-referenced report that finds surveys have been of limited use as a 

beneficiary protection tool. 

Our detailed comments on the report recimmendations are attached for your 

consideration. Thank you for the opportunity to review and comment on this report. 

Attachment 
Comments of the Health Care Financing Administration (HCFAI 

on Offrce of Insnector General (OIG) Draft Report: 

“Medicaid Managed Care: The Use of Surveys as a Beneficiarv Protection Tool.” 

(OEI-01-95-00280) 

OIG Recommendation 

HCFA should either establish a work group or technical advisory group on Medicaid 

beneficiary surveys or add surveys to the agenda of an existing group. Either group 

should provide policy-level guidance on how to make cost-effective use of beneficiary 

surveys. 

HCFA Resuonse 

We partially concur with the intent of the recommendation to provide policy-level 

guidance on how to make cost-effective use of beneficiary surveys. 

HCFA already has a Medicaid Managed Care Technical Advisory Group (MMCTAG) 

that works in conjunction with the Medicaid Managed Care Team (MMCT) in the Office 

of Managed Care (OMC). The MMCTAG is comprised of state Medicaid agency staff, 

the American Public Welfare Association, and the MMCT staff. The MMCTAG 

provides valuable input essential for the accuracy and completion of many-HCFA 

projects. As part of its fiscal year 1996 agenda, the MMCTAG formed six work groups 

to address issues they consider to be the most urgent in the rapid transition from fee-forservice to managed care. One of these groups is working on consumer information, 

which has as one of its goals the development of a document on survey instruments tested 

and validated. 

The report states beneficiary sufveys developed by state Medicaid agencies are developed 

for different purposes and to meet various objectives. We agree with that assessment. 

However, due to limited state resources, it is not feasible for states to develop surveys for 

a single purpose. When they are developed and used for a single purpose, they are ody 

one part of the overall assessment. The purposes of Medicaid surveys range from internal 

assessment of a plan’s performance, to assessing how satisfied beneficiaries are with a 

plan and its services, to identifying areas for potential improvement, etc. The purpose of 

the survey dictates how it should be constructed, what should be included, to whom and 

when it should be administered, and most importantly, how the results can and should be 

compiled and utilized. Because of the variation among states’ administration and 

coverage, it is very difficult to come up with one method or instrument that will satisfy 

everyone’s needs. Although states are currently experimenting with different instruments 

and formats, we believe they have obtained some meaningful information about 

beneficiaries (maybe not what they started out to acquire, but nonetheless, useful). 
Page 2 

We concur with the need for additional policy-level guidance on survey use. HCFA is 

already collaborating with the Agency for Health Care Policy and Research on the 

development of its Consumer Assessment of Health Plans Survey (CHAPS). This survey 

includes questions for the Medicaid population, as well as commercial and Medicare 

populations. In addition, HCFA recently prepared and distributed copies of “Customer 

Satisfaction Surveys for Medicaid Beneficiaries: A State Guide” to Medicaid Associate 

Regional Administrators for release to their respective states for input. This document, 

containing tiormation on specific survey instruments developed by states and national 

organizations, will assist states in developing or improving existing customer satisfaction 

surveys for the Medicaid population. Although we are making progress, there still needs 

to be more work done in this area, particularly on the use of beneficiary survey results. 

OIG Recommendation 

HCFA should devote greater attention to how the Medicaid agencies are using 

beneficiary surveys. It should revise its written guides for reviewing and monitoring . 

Medicaid managed care initiatives to call attention to the importance of using beneficiary 

surveys in more focused, strategic ways. 

HCFA Response 

We concur that some state surveys are very general, attempt to address too many issues at 

once, and need to be more focused. Surveys are a very important part of states’ 

requirements under the Quality Assurance Reform Initiative (QARI). As states become 

more experienced in survey design, analysis, and use, we expect the surveys to be more 

focused in what questions they are trying to answer and the purposes they are used for. 

We have, however, made some progress in this area and are working to improve it even 

further. The existing MMCTAG established a work group on consumer information, 

whose goal is to collect information on what Medicaid recipients want to know about 

health plans and providers‘and how beneficiaries think information should be presented. 

The MhKTAG also plans to work with Harvard Medical School and/or Rand in the 

piloting of the CHAPS Medicaid survey. In addition, the QARI is being replaced with a 

system that will contain quality standards for both Medicaid and Medicare. We 

anticipate additional focus on the importance of surveys and baseline data to measure 

demonstrable improvement in beneficiary care and outcomes. OMC also sponsors a 

Managed Care College each year. We plan to include a special session on survey 

development and use of survey data. Finally, as part of our continuing technical 

assistance to regional office and state Medicaid staffl we will emphasize the importance 

of surveys and how they can be used to monitor managed care initiatives. 
American Association of 

HEALTH PLAN3 

April 11, 1997 

June Gibbs Brown 

Inspector General 

Offrce of Inspector General 

Department of Health and Human Services 

330 Independence Avenue, SW 

Washington, DC 20201 

Dear Ms. Brown: 

On behalf of the American Association of Health Plans (AAHP), thank you for the opportunity to 

review and provide comments on the Office of Inspector General’s (OIG’s) draft inspection 

report, “Medicaid Managed Care: Use of Surveys as a Beneficiary Protection Tool.” AAHP 

represents approximately 1000 health plans serving over 100 million Americans nationwide. 

Many of AAHP’s members provide services to Medicaid beneficiaries. 

AAHP agrees with the basic findings and recommendations of the report. /&HP fully supports 

initiation of new efforts designed to improve information to beneficiaries about their enrollment 

options to help them make better informed c&ices. We also recognize that, for a number of 

reasons, State mandated surveys may be of limited value as a beneficiary protection tool and may 

not provide useful information to beneficiaries. 

AAHP specifically supports the recommendation that HCFA form a work group or technical 

advisory group to provide policy-level guidance on how to make cost-effective use of beneficiary 

surveys. We also appreciate and agree with the recommendation that HCFA should consider the 

relative costs of surveying beneficiaries with the relative usefulness of the survey results. 

Consideration of this issue is equally relevant whether the survey costs are incurred by the State 

Medicaid Agency directly or by the health plan. 

Our suggestions and comments are relatively ininor. We suggest several refinements to the 

Executive Summary to clarify a few issues and to give greater prominence to some important 

points made in the body of the report. The following are suggested revisions to the Executive 

summary: 

0 At the end of the second paragraph of the Background section of the Executive Summary, 

we suggest that the OIG add a sentence or two noting that there are also many other 

methods to measure plan performance, including HEDIS and the evaluation of other 

utilization data. Also, we suggest that the text note that to obtain a beneficiary 

perspective on health plans, surveys are rarely used alone and complement other types of 

performance measurement data. . 

0 In the next paragraph of the Executive Summary, we suggest that the narrative note that 

---.--- -e-(...p-.- - - - - - . ._ _ ___ __ ._ 

1129 Twentieth Street. N\V �  Suite 600 �  \Vnshineton. DfJ 2W36 �  202-778-3200 �  FAX: 202-331-7487 
June Gibbs Brown 

April 11, 1997 

Page 2 

other tools, such as descriptions of benefits, access requirements, and the delivery system, 

may assist the consumer in making informed enrollment decisions. 

0 We suggest that the Executive Summary note that there are two types of surveys. The 

first type, which is the focus of the report, is State mandated surveys (either conducted by 

the State Medicaid Agency or the health plan on its behalf). The second type of survey is 

one conducted voluntarily by health plans. We suggest that the Executive Summary note 

briefly that health plans frequently conduct their own surveys and that these surveys may 

lead to improvements by the plans. (This point is noted on page 10 of the draft report). 

AAHP also recommends that the OIG suggest that HCFA obtain the participation of health plans 

in its Workgroup activities. 

Further, we suggest that the OIG identify as an issue for consideration the possibility of giving 

health plans the discretion to decide, subject to State approval, the survey’s focus. The draft 

report suggests that survey needs may vary among health plans. If so, it would be desirable to 

give health plans the flexibility to decide-how best to meet those needs. This suggestion could be 

incorporated in the fourth bullet on page 12 of the draft report. 

Thank you again for the opportunity to comment on this draft report. If your staff would like to 

discuss further our comments, please call me at (202) 778-3209 or Michelle Fried, Counsel for 

Federal Programs, at (202) 778-8484. 

Sincerely, 

k&W . 

Executive Director, Regulatory Affairs 
APPENDIX D 

ENDNOTEB 

1. The Federal government requires Medicaid agencies enrolling their beneficiaries 

into managed care to meet certain quality assurance requirements. Among these 

requirements are those that call for each managed care plan to have an internal 

quality assurance plan, grievance procedure, and total enrollment comprised of not 

more than 75 percent Medicaid and/or Medicare beneficiaries. Federal regulations 

also require the plans to allow disenrollment on demand within 1 month of requesting 

it, although this can be waived for a 6-month lock-in period. Statutory requirements 

limit the manner in which plans can pay their physicians, generally referred to as the 

physician incentive program. Agencies must conduct periodic medical audits to ensure 

quality and access of services and provide data on reasons for beneficiary 

disenrollment. Agencies must also ensure that an independent, external review of the 

quality of services be conducted annually for each managed care plan. The agencies, 

of course, can build upon these Federal requirements, and the Health Care Financing 

Administration can waive certain requirements through the different waiver programs. 

2. C. Baldwin, “Customer Satisfaction Surveys for Medicaid Beneficiaries: A State 

Guide,” (Baltimore, MD: Office of State Health Reform Demonstrations, HCFA) 

November 1996. 

3. The 11 States are: Arizona, California, Florida, Massachusetts, Minnesota, 

Missouri, New York, Ohio, Oregon, Rhode Island, and Utah. 

4. By technical quality of care we mean treatment, examinations, follow-through, and 

preventive care as opposed to interpersonal communications and courtesy. We base 

this distinction on categories presented in the Draft Medicaid HEDIS document 

published in July of 1995. 

5. The agencies in our sample were all in various stages of the process of getting and 

using encounter data to help them assess plan performance. For the most part, as of 

this writing, agencies lack validated encounter data. They also were in various stages 

of requiring their plans to report HEDIS indicators. 

6. R. H. Brook, C. J. Kamberg, and E. A. McGlynn, “Health System Reform and 

Quality,” Journal of the American Medical Association 276(6) (August 14, 1996): 478 

and Massachusetts Healthcare Purchaser Group, “Cost/Quality Challenge,” March 

1996. 

7. In Oregon, the State agency conducted a baseline survey in 1994, then conducted 

another survey in 1996. The agency revised the second survey, thereby affecting the 

direct comparability of results between the two surveys. (The Oregon Health Plan 

D-l 
Medicaid Demonstration Project: 1994 and 1996 Comparative Client Satisfaction Data, 

Preliminary Aggregate Report.) 

8. Minnesota Health Data Institute, “Evaluation Report of the Minnesota Health Data 

Institute 1995 Consumer Survey,” (July 1996) 

9. Joseph Mangano, Ed., “Oregon Consumers Favor Doctor Data” Quality 

Management Update 6(18) (September 18, 1996). 

10. Robert Cunningham, Ed., “Consumers Gaining Leverage in Quality Assurance 

Arena,” Medicine & Health (June 17, 1996). See also Harold S. Luft, “Modifying 

managed Competition to Address Cost and Quality,” Health Affairs (Spring 1996): 23-

28 and Frederick Schneider Research, “Medicaid and Managed Care: Focus Group 

Studies of Low Income Medicaid Beneficiaries in Five States,” (Washington DC: May 

1996). 

11. Henry J. Kaiser Family Foundation, “Americans as Health Care Consumers: The 

Role of Quality Information,” (October 1996) and Robert Cunningham, Ed., 

“Consumers Gaining Leverage in Quality Assessment Arena.” 

12. The Medicaid HEDIS (Health Plan Employer Data and Information Set) Work 

Group comprises individuals and organizations that worked together to develop 

measures for monitoring the performance of managed care plans serving the Medicaid 

population. The National Committee for Quality Assurance sponsored the Work 

Group in association with HCFA and the American Public Welfare Association. 

13. The Agency for Health Care Policy and Research (AHCPR) is currently funding a 

5-year project aimed, in part, at developing and testing questionnaires to assess plans. 

This project, called the Consumer Assessment of Health Plans Study (CAHPS), is at 

year 3 and beginning to field test its questionnaires. The CAHPS encompasses all 

types of health plans as well as the privately and publicly (Medicare and Medicaid) 

insured. The Health Care Financing Administration has called for managed care 

plans enrolling Medicare beneficiaries to use the CAHPS questionnaire. Eventually 

the AHCPR expects that its CAHPS questionnaires will be available to interested 

parties without charge. 

14. The response rate was calculated after deducting 204 beneficiaries identified as 

ineligible for reasons such as disconnected phones, moved out of the area, and 

deceased. 

15. Of the 17 respondents that indicated they had not conducted any surveys that 

included Medicaid beneficiaries in the past 2 years, 14 had plans to conduct such 

surveys within the next year. 

16. In another recent survey of health plans (with Medicare risk and cost contracts), 

the Office of Inspector General also found great variety in plan approaches to surveys 

(Office of Inspector General, HMO Customer Satisfaction Suweys, March 1996). 

D-2 
17. In another recent survey of health plans (with Medicare risk and cost contracts), 

the Office of Inspector General found that plans used their survey results as much for 

marketing as for improvements. (Office of Inspector General, HMO Customer 

Satisfaction Suweys, March 1996) 

18. American Association of Health Plans, “1995 AAHP HMO PPO Trends Report.” 

19. The CAHPS aims to help consumers choose the best-suited health plan for their 

needs by developing and testing questionnaires to assess plans, reporting the survey 

information to consumers, and evaluating their usefulness. This ambitious project 

encompasses the privately insured as well as the Medicaid and Medicare populations. 

As of this writing, the CAHPS project is at year 3 and beginning to field test its 

surveys. 

20. HCFA, “Monitoring Guide for Medicaid Managed Care Initiatives,” Working 

Draft (September 1996) and “Medicaid Pre-Implementation Review Guide for Section 

1115 Demonstration Waivers,” Working Draft, (August 1995). 

21. The 1981 amendments also allow States to enroll Medicaid beneficiaries in limitedrisk managed care organizations (i.e., no risk for inpatient care) that fail to meet 

Federal qualifications. 

22. The following States have implemented their 1115 waivers: Alabama, Arizona, 

California, Delaware, Hawaii, Minnesota, Ohio, Oklahoma, Oregon, Rhode Island, 

Tennessee, and Vermont. The following States have approved 1115 waivers that are 

pending implementation: Florida, Illinois, Kentucky, Maryland, and Massachusetts. 

South Carolina has an approved waiver but no plans for implementation. 

23. States with 1115 waivers under review are: Georgia, Kansas, Louisiana, Missouri, 

New Hampshire, New York, Texas, Utah, and Washington. 

24. Federal regulations define comprehensive services as either inpatient hospital 

services and one other mandatory service or three or more mandatory services 

(42 CFR 434.21). Mandatory services are defined in statute as inpatient and 

outpatient care, physicians’ services, and laboratory and diagnostic services, among 

others (42 USC §1396d(a)). 

D-3