Medicare: Communications to Beneficiaries on the Prescription Drug Benefit Could Be Improved
Released: 2006-05-03T00:00:00

Letter

Conclusions

Within the past 6 months, millions of Medicare beneficiaries have been making important decisions about their prescription drug coverage and have needed access to information about the new Part D benefit to make appropriate choices. CMS faced a tremendous challenge in responding to this need and, within short time frames, developed a range of outreach and educational materials to inform beneficiaries and their advisers about Part D. To disseminate these materials, CMS largely added information to existing resources, including written documents, such as Medicare & You; the 1-800-MEDICARE help line; the Medicare Web site; and support for SHIPs. However, CMS has not ensured that its communications to beneficiaries and their advisers are provided in a manner that is consistently clear, complete, accurate, and usable. Six months have passed since these materials were first made available to beneficiaries, and their limitations could result in confusion among those seeking to make coverage decisions. Although the initial enrollment period for Part D will end on May 15, 2006, CMS will continue to play a pivotal role in providing beneficiaries with information about the drug benefit during the year and in subsequent enrollment periods. CMS has an opportunity to enhance its communications on the Part D benefit. This would allow beneficiaries and their advisers to be better prepared when deciding whether to enroll in the benefit, and if enrolling, which drug plan to choose.

Recommendations for Executive Action

In order to improve the Part D benefit education and outreach materials that CMS provides to Medicare beneficiaries, we are recommending that the CMS Administrator take the following four actions: Ensure that CMS’s written documents describe the Part D benefit in a manner that is consistent with commonly recognized communications guidelines and that is responsive to the intended audience’s needs.

Determine why CSRs frequently do not search for available drug plans if the caller does not provide personal identifying information.

Monitor the accuracy and completeness of CSRs’ responses to callers’ inquiries and identify tools targeted to improve their performance in responding to questions concerning the Part D benefit, such as additional scripts and training. Improve the usability of the Part D portion of the Medicare Web site by refining Web-based tools, providing workable site navigation features and links, and making Web-based forms easier to use and correct.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from CMS (see app. III). CMS said that it did not believe our findings presented a complete and accurate picture of its Part D communications activities. CMS discussed several concerns regarding our findings on its written documents and the 1-800-MEDICARE help line. However, CMS did not disagree with our findings regarding the Medicare Web site or the role of SHIPs. CMS also said that it supports the goals of our recommendations and is already taking steps to implement them, such as continually enhancing and refining its Web-based tools.

CMS discussed concerns regarding the completeness and accuracy of our findings in terms of activities we did not examine, as well as those we did. CMS stated that our findings were not complete because our report did not examine all of the agency’s efforts to educate Medicare beneficiaries and specifically mentioned that we did not examine the broad array of communication tools it has made available, including the development of its network of grassroots partners throughout the country. We recognize that CMS has taken advantage of many vehicles to communicate with beneficiaries and their advisers. However, we focused our work on the four specific mechanisms that we believed would have the greatest impact on beneficiaries—written materials, the 1-800-MEDICARE help line, the Medicare Web site, and the SHIPs. In addition, CMS stated that our report is based on information from January and February 2006, and that it has undertaken a number of activities since then to address the problems we identified. Although we appreciate CMS’s efforts to improve its Part D communications to beneficiaries on an ongoing basis, we believe it is unlikely that the problems we identified in this report could have been corrected yet given their nature and scope.

CMS raised two concerns with our examination of a sample of written materials. First, it criticized our use of readability tests to assess the clarity of the six sample documents we reviewed. For example, CMS said that common multisyllabic words would inappropriately inflate the reading level. However, we found that reading levels remained high after adjusting for 26 multisyllabic words a Medicare beneficiary would encounter, such as Social Security Administration. CMS also pointed out that some experts find such assessments to be misleading. Because we recognize that there is some controversy surrounding the use of reading levels, we included two additional assessments to supplement this readability analysis—the assessment of design and organization of the sample documents based on 60 commonly recognized communications guidelines and an examination of the usability of six sample documents, involving 11 beneficiaries and 5 advisers.

Second, CMS expressed concern about our examination of the usability of the six sample documents. The participating beneficiaries and advisers were called on to perform 18 specified tasks, after reading the selected materials, including a section of the Medicare & You handbook. CMS suggested that the task asking beneficiaries and advisers to calculate their out-of-pocket drug costs was inappropriate because there are many other tools that can be used to more effectively compare costs. We do not disagree with CMS that there are a number of ways beneficiaries may complete this calculation; however, we nonetheless believe that it is important that beneficiaries be able to complete this task on the basis of reading Medicare & You, which, as CMS points out, is widely disseminated to beneficiaries, reaching all beneficiary households each year. In addition, CMS noted that it was not able to examine our detailed methodology regarding the clarity of written materials—including assessments performed by one of our contractors concerning readability and document design and organization. We plan to share this information with CMS, once our report has become public.

Finally, CMS took issue with one aspect of our evaluation of the 1-800-MEDICARE help line. Specifically, CMS said the 41 percent accuracy rate associated with one of the five questions we asked was misleading, because, according to CMS, we failed to analyze 35 of the 100 responses. However, we disagree. This question addressed which drug plan would cost the least for a beneficiary with certain specified prescription drug needs. We analyzed these 35 responses to this question and found the responses to be inappropriate. The CSRs would not provide us with the information we were seeking because we did not supply personal identifying information, such as the beneficiary’s Medicare number or date of birth. We considered such responses inappropriate because the CSRs could have answered this question without personal identifying information by using CMS’s Web-based prescription drug plan finder tool. Although CMS said that it has emphasized to CSRs, through training and broadcast messages, that it is permissible to provide the information we requested without requiring information that would personally identify a beneficiary, in these 35 instances, the CSR simply told us that our question could not be answered. CMS also said that the bulk of these inappropriate responses were related to our request that the CSR use only brand-name drugs. This is incorrect—none of these 35 responses were considered incorrect or inappropriate because of a request that the CSR use only brand-name drugs—as that was not part of our question.

As arranged with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days after its date. At that time, we will send copies of this report to the Secretary of Health and Human Services, the Administrator of the Centers for Medicare & Medicaid Services, and other interested parties. We will also make copies available to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (312) 220-7600 or aronovitzl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Briefing on Medicare Part D

Purpose and Objectives

for overseeing this new benefit. CMS has taken steps to inform beneficiaries and their advisers about Part D using written documents, a toll-free help line, and an Internet Web site. CMS also gives State Health Insurance Assistance Programs (SHIP) funds to provide information about the Medicare program, including Part D.

Purpose and Objectives (continued)

Objective 1: Written Documents Methodology

sample of six CMS documents describing the Part D benefit. The sample was selected to represent a variety of document types, content, and target audiences.

Medicare & You (Section 6: Medicare Prescription Drug Coverage)

Institutes for Research (AIR) to assess the clarity of sample documents.

Dual-eligible beneficiaries are Medicare beneficiaries who receive full Medicaid benefits for services not covered by Medicare.

Medicare Advantage replaces the Medicare+Choice managed care program and expands the availability of private health plan options to Medicare beneficiaries.

Do You Have a Medigap Policywith Prescription Drug Coverage? sold by private insurers to help pay for Medicare cost- sharing requirements, as well as for some services not provided by Medicare.

Objective 1: Written Documents Methodology (continued)

describing the Part D benefit, AIR

Objective 1: Written Documents Methodology (continued)

tested the usability of sample documents with 16 participants—11 Medicare beneficiaries, including 1 disabled beneficiary who was under 65, and 5 advisers to beneficiaries.

Everyone was asked to perform 18 specified tasks related to enrollment, coverage, costs, penalty, and informational resources. They were also asked to provide feedback about their experiences.

Although the size of the group was small, research shows that as few as 5 individuals can provide meaningful insights into common problems.

Objective 1: Written Documents Methodology (continued)

To evaluate accuracy, we reviewed the sample documents for

consistency with laws, regulations, and CMS guidance.

Objective 1: Written Documents—Documents Lack Clarity

Readability assessment: Sample documents explaining the Part D

benefit are written at a reading level that is difficult for many seniors.

Reading levels for the sample documents were challenging for at least the 40 percent of seniors, who read at or below the 5th grade level.

Reading level estimates for the sample texts1 ranged from 7th grade to postcollege level.

Reading levels remain challenging for at least 40 percent of seniors even after adjusting for 26 multisyllabic words, such as Medicare, Medicare Advantage, and Social Security Administration. After the adjustment, the estimated reading level ranged from 8th to 12th grade.

Estimates have a likely margin of error of ± two grades.

Objective 1: Written Documents—Clarity (continued)

Document design and organization assessment: The sample

documents demonstrated adherence to about half of the 60 commonly recognized written communications guidelines, on average.

Desirable features: The documents were written with a respectful and polite tone, were free of clichés and slang, contained useful contact information, included concise and descriptive headings, and generally followed graphic and formatting guidelines.

Objective 1: Written Documents—Clarity (continued)

Undesirable features: The documents used too much technical jargon, often did not define difficult terms, included sentences and some paragraphs that were too long, did not use sufficient summaries to assist the reader in identifying key points.

Objective 1: Written Documents—Clarity (continued)

Usability assessment: Beneficiaries and advisers to beneficiaries

were frustrated by the documents’ lack of clarity and often could not complete the 18 assigned tasks.

One of the 18 assigned tasks was completed by all beneficiaries and advisers.

Eleven of the 18 assigned tasks were completed by at least half of the beneficiaries and advisers.

Four of the 18 assigned tasks were completed by 2 or fewer of the 11 beneficiaries.

Nine of the 18 assigned tasks, were completed by 2 or fewer of the 5 advisers.

Objective 1: Written Documents—Clarity (continued)

Some of the tasks that proved difficult included

computing projected total out-of-pocket costs for a plan that provided Part D’s standard coverage (successfully completed by none of the 11 beneficiaries and 2 of the 5 advisers), evaluating whether it was possible to enroll in Medicare Part D and keep drug coverage from a retiree health plan (successfully completed by 2 beneficiaries and 2 advisers), and determining the course of action for dual-eligibles who are automatically enrolled in a plan that does not cover all drugs used (successfully completed by 4 beneficiaries and 1 adviser).

Objective 1: Written Documents—Clarity (continued)

Participants described documents as too wordy, confusing, and hard

to follow.

Participants struggled with technical terms, such as “classes of commonly prescribed drugs” and “formulary,” which is a list of drugs covered by a plan.

Even when most participants were able to complete the tasks, they expressed confusion and frustration.

Objective 1: Written Documents—Documents Are Generally Complete

All documents reviewed provided sources of assistance and

relevant contact information, which could aid in identifying next steps for coverage decisions.

All documents reviewed provided the dates of the start of initial program enrollment and coverage.

Objective 1: Written Documents— Completeness (continued)

The documents did not provide sufficient information about the

cumulative effect of the penalty for missing the initial enrollment deadline.

Objective 1: Written Documents—Documents Are Generally Accurate

Our analysis showed that the sample documents were generally

accurate and that the text was consistent with MMA, implementing regulations, and agency guidance.

However, we noted a few misleading statements in Medicare & You.

The document implied that if a beneficiary’s doctor applied for an exception it would be granted, whereas exceptions to the formulary are granted at each plan sponsor’s discretion.

The document outlined the minimum requirements for standard coverage by Part D plans. However, it did not indicate that few plans offer this exact coverage and that beneficiaries should be prepared to compare plans with varying premiums, co-payments, and covered drugs to choose plans that best suit them.

Objective 2: 1-800-MEDICARE Help Line Methodology

different days of the week from January 17 to February 7, 2006, to match the daily and hourly pattern of calls reported by 1-800-MEDICARE in October 2005.

Objective 2: 1-800-MEDICARE Help Line Methodology (continued)

To evaluate the accuracy and completeness of CSRs’ responses to

our five questions, we used three resources: the prescription drug finder tool on the Medicare Web site, the 1-800-MEDICARE scripts prepared by CMS and contractors for CSRs to use in responding to callers’ questions, and input from CMS officials on the criteria we used to evaluate responses.

To evaluate the promptness of the help line in answering calls, we recorded the length of time it took to connect to a CSR for each call.

Objective 2: 1-800-MEDICARE Help Line Methodology (continued)

Objective 2: 1-800-MEDICARE Help Line Methodology (continued)

Question

1. What drug plan can a beneficiary get that will cover all of his/her drugs at a pharmacy; have a mail-order option; and cost the least amount annually with [or without] a deductible?

An accurate and complete response would identify the prescription drug plan that has the lowest estimated annual cost for the drugs the beneficiary uses.

2. Can a beneficiary who is in a nursing home and not on Medicaid sign up for a prescription drug plan?

An accurate and complete response would indicate that such a beneficiary can choose whether to enroll in a Medicare prescription drug plan.

Objective 2: 1-800-MEDICARE Help Line Methodology (continued)

Question

3. Can a beneficiary enroll in the Medicare prescription drug program and keep his/her current Medigap policy?

An accurate and complete response would inform the caller that enrolling for the prescription drug benefit would depend on whether the beneficiary’s Medigap plan was creditable—that is, whether the coverage it provided was at least as good as Medicare’s standard prescription drug coverage—or noncreditable. The CSR response would also mention that the beneficiary’s Medigap plan should have sent him/her information that outlines options.

Objective 2: 1-800-MEDICARE Help Line Methodology (continued)

Question

4. What options does a beneficiary, who has retiree health insurance with prescription drug coverage that is not as good as the Medicare prescription drug coverage, have as it relates to the Medicare benefit?

An accurate and complete response would indicate that a beneficiary has two options: (1) keep current health plan and join the prescription drug plan later with a penalty; or (2) drop current coverage and join a Medicare drug plan.

5. How do I know if a beneficiary qualifies for extra help?

An accurate and complete response would refer the beneficiary to the Social Security Administration.

Objective 2: 1-800-MEDICARE Responses Often Accurate and Complete, but Some Not

We found that the 1-800-

MEDICARE help line provided accurate and complete answers to 334 of our 500 calls, a rate of about 67 percent. In addition, it provided accurate—but incomplete— answers for about 3 percent of our calls.

Objective 2: 1-800-MEDICARE—Variation in Results for Individual Questions

The accuracy and

completeness of responses to our five questions varied significantly, from 41 percent to 90 percent.

Average for all

Objective 2: 1-800-MEDICARE—Variation (continued)

CSRs answered some questions better than others. For example:

CSRs accurately and completely answered question 5 (whether a beneficiary qualifies for extra help), which had a specific script, 90 percent of the time.

CSRs accurately and completely answered question 2 (whether a beneficiary in a nursing home, who was not on Medicaid, could sign up for the drug benefit) 79 percent of the time—even though there was no specific script for the question.

Objective 2: 1-800-MEDICARE—Variation (continued)

CSRs’ responses for question 3 (whether a beneficiary with a Medigap policy could enroll in the drug benefit) were accurate and complete 66 percent of the time. Many of the responses were inaccurate because they did not provide adequate information about creditable and noncreditable coverage.

The accuracy and completeness rate for question 4 (about retiree health insurance) was 58 percent. Many of the responses were inaccurate because the CSRs did not follow the available script or provide sufficient information about the implications of the beneficiary's decision.

Objective 2: 1-800-MEDICARE—Variation (continued)

CSRs’ responses to question 1 (which requires CSRs to use the prescription drug plan finder Web tool) were accurate and complete less than 50 percent of the time. The rate is largely caused by CSRs’ inappropriate responses—35 out of 100 times— that they were unable to answer the question without personal identifying information, such as the beneficiary’s Medicare number or date of birth.

Objective 2: 1-800-MEDICARE—Variation (continued)

Unintentional disconnections occurred when the system

inadvertently disconnected the call (19 calls).

Intentional disconnections were programmed by the telephone company when wait times were projected to exceed 20 minutes (3 calls).

The prescription drug plan finder Web tool used by CSRs was not operative at the time of our call (1 call).

Objective 2: 1-800-MEDICARE—Variation in Wait Times

The amount of time we had to wait to speak with a CSR varied

significantly, ranging from no wait to more than 55 minutes.

About 75 percent of calls were connected in less than 5 minutes.

For calls where we waited more than 5 minutes to speak to a CSR, the wait time ranged from 5 minutes to over 55 minutes.

Sixty-two calls were on hold from 5 to 14 minutes, 59 seconds. Thirty-nine calls were on hold from 15 to 24 minutes, 59 seconds.

Twenty-five calls were on hold 25 minutes or more.

For both intentional and unintentional disconnections, we often waited more than 5 minutes before the disconnection occurred. In one case, we were placed on hold for 54 minutes before being disconnected.

Objective 3: Medicare Web Site Methodology

Web site, we contracted with Nielsen Norman Group (NN/g), a firm with expertise in Web design.

Objective 3: Medicare Web Site Methodology (continued)

Section 508 requires that all federal Web sites be designed to

make information and services fully available to individuals with disabilities.

Our review included an examination of CMS’s March 2006 report assessing the compliance of its Medicare Web site with this federal requirement and discussions with CMS officials.

Objective 3: Medicare Web Site Methodology (continued)

NN/g performed the following three separate evaluations: Evaluation one: NN/g calculated an overall score of the site’s usability, to reflect the ease of finding necessary information and performing various tasks. For this calculation, NN/g considered various factors, such as site navigation, customer support, and presentation of online forms.

Objective 3: Medicare Web Site Methodology (continued)

Evaluation two: NN/g evaluated in detail the usability of 137 detailed aspects of the Part D benefit portion of the Web site. Topics included Web design (e.g., home page, navigation, search function, graphics, and organization); tools (e.g., plan finder); writing style (e.g., tone, content, legibility, and readability); accessibility (e.g., availability of site version for the blind); and languages (e.g., links for users who have difficulty reading English).

Objective 3: Medicare Web Site Methodology (continued)

Evaluation three: NN/g conducted a total of 34 user tests to determine the ease of performing a variety of Web-related tasks, such as browsing the site, making a change in address, finding plan information under certain scenarios, comparing Medigap and Part D drug coverage, and determining how to join a plan.

NN/g asked five Medicare beneficiaries—who were not disabled—and two advisers to beneficiaries to perform one or more user tests each using the Web site.

At the end of the user tests, the seven participants were asked to provide feedback about their experiences.

Objective 3: Medicare Web Site Difficult to Use

for improvement. The usability score was 47 percent for seniors and 53 percent for younger adults. While there is no widely accepted benchmark for usability, these scores indicate that using the site can be difficult. For example, tools such as the drug plan finder were complicated to use, and forms that collect information online from users were difficult to correct if the user made an error.

Objective 3: Medicare Web Site Difficult to Use (continued)

Objective 3: Medicare Web Site Difficult to Use (continued)

information to assist navigation was often not helpful—for example, text labels associated with links were not always functioning; and the writing style presented some challenges—for example, material was written at the 11th grade level.

Objective 3: Medicare Web Site Difficult to Use (continued)

For evaluation three, the 34 user tests showed that the site was a challenge for the seven participants to use. For example: For 12 of the 34 tests, participants’ initial reactions were that they would not be able to complete the tests and wanted to quit trying.

On average, participants were able to proceed slightly more than halfway through each of the 34 tests.

When asked for feedback on their experience with using the site, the seven participants, on average, indicated high frustration levels and low satisfaction.

Objective 3: Medicare Web Site Difficult to Use (continued)

established features to make information on its Medicare Web site accessible to disabled users. For example, CMS provides a “screen reader” version of the site for the visually impaired. This technology translates text and data into spoken words.

Objective 3: Medicare Web Site Difficult to Use (continued)

CMS’s March 2006 review of its site’s compliance with section 508

showed that two requirements were not met: The plan finder did not provide alternative text for all images—that is, there was no text for the screen reader to read. Therefore, images could not be translated into spoken words for the visually impaired.

The plan finder did not allow screen readers to recognize form fields and translate forms into spoken words. As a result, visually impaired users would not have been able to complete Web-based forms.

A CMS official told us that the agency made the necessary corrections on April 20, 2006, but we did not verify that these corrections were made.

Objective 4: SHIP Methodology

Objective 4: SHIPs’ Responses to Beneficiaries’ Needs Concerning Part D

the Part D benefit, CMS increased SHIP funding in recent years. Funding for the 2003 SHIP grant year2 was $12 million, and it reached $31.7 million for the 2005 grant year. CMS kept funding relatively high for the 2006 grant year—$30 million—to ensure that SHIPs continue to play an important role in educating beneficiaries about the Part D benefit.

A SHIP grant year begins on April 1 of the year the funds become available.

Objective 4: SHIPs’ Responses to Part D (continued)

million people. According to preliminary data for the first 9 months of the 2005 SHIP grant year—when CMS was gearing up its outreach and education on Part D—SHIPs served approximately 3.3 million individuals, an increase of nearly 770,000 from the prior full grant year. CMS attributes this increase in demand for services to beneficiaries’ need for assistance on the Part D benefit.

Objective 4: SHIPs’ Responses to Part D (continued)

The average number of calls referred from the 1-800-MEDICARE

help line to SHIPs has increased significantly.

The monthly average of number of calls referred to SHIPs increased from 16,000 referrals for May through September 2005 to approximately 43,000 for October and November 2005, the months around the time when enrollment in the Part D benefit began.

According to CMS officials, this increased demand was influenced by callers seeking advice on choosing a drug plan. Unlike CSRs on the help line, SHIP counselors can offer individualized guidance to callers.

Objective 4: SHIPs’ Responses to Part D (continued)

California served over 120,000 people in January 2006, compared

to about 35,000 served in all of 2005.

Florida, mostly during November and December of 2005, held at least six “phone bank” events—where SHIP counselors were available to take calls on the Part D benefit during live newscasts. Florida plans to hold two additional phone banks as the May 15 enrollment deadline approaches.

Objective 4: SHIPs’ Responses to Part D (continued)

New York reported nearly doubling its formal training sessions for SHIP counselors in 2005, to prepare them for the demand for services related to the Part D benefit.

Texas counseled 45,719 clients and conducted 523 outreach events from November 15, 2005—the official start of the enrollment period—to March 22, 2006.

Pennsylvania held over 3,000 enrollment events, which were attended by more than 130,000 people, from May 2005 to February 28, 2006.

Objective 4: SHIPs’ Responses to Part D (continued)

Since December 2005, CMS has been conducting biweekly

meetings with its regional offices, which interact directly with SHIP offices, to gauge SHIPs’ ability to meet the demands of beneficiaries.

Appendix II: Objectives, Scope, and Methodology

In this report, we assessed (1) the extent to which the Centers for Medicare & Medicaid Services’ (CMS) written documents describe the Medicare Part D prescription drug benefit in a clear, complete, and accurate manner; (2) the effectiveness of CMS’s 1-800-MEDICARE help line in providing accurate, complete, and prompt responses to callers inquiring about the Part D benefit; (3) whether CMS’s Medicare Web site presents information on the Part D benefit in a usable manner; and (4) how CMS has used State Health Insurance Assistance Programs (SHIP) to respond to the needs of Medicare beneficiaries for information on the Part D benefit. To obtain information on CMS’s efforts to educate beneficiaries about Part D, we interviewed agency officials responsible for Part D written documents, the 1-800-MEDICARE help line, the Medicare Web site, and SHIPs. Following our briefing of congressional staff on April 19, 2006, the briefing slides were updated to reflect CMS’s reported correction to the Medicare Web site to comply with section 508 of the Rehabilitation Act of 1973. We determined that the data used were sufficiently reliable for the purposes of this report.

Written Documents

To assess the clarity, completeness, and accuracy of written documents, we compiled a list of all available CMS-issued Part D benefit publications intended to inform beneficiaries and their advisers and selected a sample of 6 from the 70 CMS documents available, as of December 7, 2005, for in- depth review, as shown in table 1. The sample Part D documents were chosen to represent a variety of publication types, such as frequently asked questions and fact sheets available to beneficiaries about the Part D drug benefit. We selected documents that targeted all beneficiaries or those with unique drug coverage concerns, such as dual-eligibles and beneficiaries with Medigap.

To evaluate clarity, we contracted with the American Institutes for Research (AIR)—a firm with experience in evaluating written material. AIR evaluated the texts of the six sample documents using three methodologies: 1. three standard readability tests; 2. 60 commonly recognized written communications guidelines, including practices to aid senior readers; and 3. user testing with 11 Medicare beneficiaries and 5 advisers to beneficiaries, who performed 18 specified tasks related to enrollment, coverage, cost, penalty, and information resources and provided feedback about their experiences.

We reviewed the sample documents for completeness to determine whether they contained sufficient information to allow the beneficiaries to identify (1) their next steps in determining whether to enroll and what plan to choose and (2) important factors, such as penalty provisions, that could affect their coverage decisions. To identify those important factors associated with the Part D benefit, we reviewed relevant laws, regulations, and 1-800-MEDICARE scripts prepared for customer service representatives (CSR) to read to callers and obtained information from advocacy groups. To evaluate the accuracy of information, we reviewed the sample materials for compliance with laws, regulations, and CMS guidance.

The 1-800-MEDICARE Help Line

To determine the accuracy and completeness of information provided regarding the Part D benefit, we placed a total of 500 calls to the 1-800- MEDICARE help line. We posed one of five questions about Part D in each call, so that each question was asked 100 times. Each question was pretested before we finalized its wording. We randomly placed calls at different times of the day and different days of the week from January 17 to February 7, 2006. Our calling times were chosen to match the daily and hourly pattern of calls reported by 1-800-MEDICARE in October 2005. We informed CMS officials that we would be placing calls; however, we did not tell them the questions we would ask or the specific dates and times that we would be placing our calls.

To select the five questions, we considered topics identified in the Medicare Web site’s frequently asked questions. In addition, we considered topics most frequently addressed by 1-800-MEDICARE CSRs based on help line reports. To evaluate the accuracy of CSRs’ responses to our five questions, we used three resources: (1) the prescription drug plan finder tool on the Medicare Web site, (2) 1-800-MEDICARE scripts, and (3) input obtained from CMS officials on the criteria we used for evaluating CSR responses. Table 2 lists the questions we asked and the criteria we used to evaluate the accuracy of responses.

When placing our calls, we identified ourselves as a beneficiary’s relative, but did not provide CSRs with specific identifying information, such as a Medicare beneficiary number or date of birth. During our calls, CSRs were not aware that their responses would be included in a research study. We recorded the length of each call, including wait times, and the time it took before being connected to a CSR. We evaluated the accuracy and completeness of the responses by CSRs to the 500 calls by determining whether key information was provided.

The results from our 500 calls are limited to those calls and are not generalizable to the universe of calls made to the help line. The questions we asked were limited to matters concerning the Part D benefit and do not encompass all of the questions callers might ask.

Medicare Web Site

We contracted with the Nielsen Norman Group (NN/g)—a firm with expertise in Web design—to assess the usability of the Part D information available on the Medicare Web site. This study consisted of three separate evaluations. First, NN/g compared the site’s compliance with established usability guidelines to determine a usability score to reflect the ease of finding necessary information and performing various tasks. Specifically, to determine the usability scores, NN/g evaluated various aspects of the Web site using industry-recognized “good” Web design practices, as indicated by the contractor, and the collective body of knowledge from NN/g internal reports and experts, or NN/g usability guidelines.

Second, NN/g determined the degree of difficulty associated with 137 detailed aspects of Web site design for the Part D portion of the site. The 137 aspects fall into the following general categories: overall Web design (e.g., home page, navigation, search function, graphics, and overall organization); tools (e.g., plan finder); writing style (e.g., content, tone, legibility, and readability); accessibility (e.g., availability of a version of the Web site for the blind); and languages (e.g., availability of languages other than English).

NN/g determined the difficulty level in using each of the 137 aspects. NN/g noted aspects that had good design and would not be expected to cause confusion. For those aspects with a design that would be expected to cause confusion, NN/g ranked the associated difficulty level as high, medium, or low.

Third, NN/g performed a qualitative evaluation on January 20 and 23, 2006, to test the ability of five Medicare beneficiaries and two beneficiary advisers to perform specified tasks related to Medicare beneficiaries using the Web site and to obtain feedback about participants’ experiences. While the results are not statistically valid, these users provided important insights into the usability of the Medicare Web site. Participants were asked to “think out loud” as they worked through their tasks, while an NN/g facilitator observed their behavior and took notes. NN/g gave each task a score. At the end of their sessions, NN/g asked participants for input regarding their confidence in the answers they obtained from the Web site, and their overall satisfaction and frustration levels associated with using the site.

Finally, we obtained the results of CMS’s March 2006 review of its Web site’s compliance with section 508 of the Rehabilitation Act of 1973, as amended. This law requires federal agencies to make the information on their Web sites accessible to people with disabilities. We also discussed the results of this review with agency officials and followed up with them to determine the status of CMS’s corrective actions.

State Health Insurance Assistance Programs

To determine the role of SHIPs in helping Medicare beneficiaries understand Part D, we interviewed CMS officials who monitor SHIPs’ activities. We also reviewed information that we obtained from CMS officials and other sources on the program, its funding, changes made in response to the introduction of Part D, and the impact of Part D on the demand for SHIP services. In addition, we interviewed SHIP officials in California, Florida, New York, Texas, and Pennsylvania—the five states with the largest Medicare populations—to obtain information on the experience of their SHIPs with Part D.

We conducted our work from November 2005 through May 2006 in accordance with generally accepted government auditing standards.

Appendix III: Comments from the Centers for Medicare & Medicaid Services

Appendix IV: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Susan T. Anthony and Geraldine Redican-Bigott, Assistant Directors; Ramsey L. Asaly; Enchelle Bolden; Laura Brogan; Shaunessye D. Curry; Chir-Jen Huang; M. Peter Juang; Ba Lin; Michaela M. Monaghan; Roseanne Price; Pauline Seretakis; Margaret J. Weber; and Craig H. Winslow made contributions to this report.