Medicare: Communications to Beneficiaries on the Prescription Drug Benefit Could Be Improved
Published: 2006-05-03T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: On January 1, 2006, Medicare began providing coverage for outpatient prescription drugs through its new Part D benefit. Beneficiaries who enroll in Part D may choose a drug plan from those offered by private plan sponsors under contract to the Centers for Medicare & Medicaid Services (CMS), which administers the Part D benefit. Beneficiaries have until May 15, 2006, to enroll in the Part D benefit and select a plan without the risk of penalties. GAO was asked to review the quality of CMS's communications on the Part D benefit. GAO examined 70 CMS publications to select 6 documents for review and contracted with the American Institutes for Research to evaluate the clarity of these texts; made 500 calls to the 1-800-MEDICARE help line; and contracted with the Nielsen Norman Group to evaluate the usability of the Medicare Web site.
section_title: What GAO Found
paragraphs: The information given in the six sample documents that GAO reviewed describing the Part D benefit was largely complete and accurate, although this information lacked clarity. The documents were unclear in two ways. First, although about 40 percent of seniors read at or below the fifth-grade level, the reading levels of these documents ranged from seventh grade to postcollege. Second, on average, the six documents did not comply with about half of 60 common guidelines for good communication. For example, the documents used too much technical jargon and often did not define difficult terms, such as formulary. Moreover, 16 beneficiaries and advisers that GAO tested reported frustration with the documents' lack of clarity and had difficulty completing the tasks assigned to them. Although the documents lacked clarity, they informed readers of enrollment steps and factors affecting coverage decisions and were consistent with laws, regulations, and agency guidance. Customer service representatives (CSR) responded to the 500 calls GAO placed to CMS's 1-800-MEDICARE help line accurately and completely about two-thirds of the time. Of the remainder, 18 percent of the calls received inaccurate responses, 8 percent of the responses were inappropriate given the question asked, and about 3 percent received incomplete responses. In addition, about 5 percent of GAO's calls were not answered, primarily because of disconnections. Accuracy and completeness rates of CSRs' responses varied significantly across the five questions GAO asked. For example, while CSRs provided accurate and complete responses to calls about beneficiaries' eligibility for extra help 90 percent of the time, the accuracy rate for calls concerning the drug plan that would cost the least for a specified beneficiary was 41 percent. For this question, the CSRs responded inappropriately for 35 percent of the calls by explaining that they could not identify the least costly plan without the beneficiary's personal information--even though CSRs had the information needed to answer the question. The time GAO callers waited to speak with CSRs also varied, ranging from no wait time to over 55 minutes. For 75 percent of the calls--374 of the 500--the wait was less than 5 minutes. The Part D benefit portion of the Medicare Web site can be difficult to use. GAO's test of the site's overall usability--the ease of finding needed information and performing various tasks--resulted in scores of 47 percent for seniors and 53 percent for younger adults, out of a possible 100 percent. While there is no widely accepted benchmark for usability, these scores indicate that using the site can be difficult. For example, the prescription drug plan finder was complicated to use and some of its key functions, such as "continue" and "choose a drug plan," were often not visible on the page without scrolling down.

section_title: Letter
paragraphs: In the most significant change to the Medicare program since its inception,  the Medicare Prescription Drug, Improvement, and Modernization Act of  2003 (MMA) established an outpatient prescription drug benefit in  Medicare, known as the Part D benefit. Coverage for this new benefit  began on January 1, 2006. Until this time, Medicare, the federal program  that finances health care benefits for about 42 million elderly and disabled  beneficiaries, had not generally provided coverage for outpatient  prescription drugs. Beneficiaries may choose a Part D plan from multiple  plans offered by private sponsors under contract to the Centers for  Medicare & Medicaid Services (CMS), the agency that is responsible for  administering the Medicare program, including the Part D benefit. These  plans differ in the drugs they cover and the pharmacies they use. In  addition, the costs to the enrollee for the monthly premium, the annual  deductible, and co-payments for covered drugs vary by plan. As of   April 20, 2006, more than 30 million of Medicare’s 42 million beneficiaries  were enrolled in a Part D plan or had other outpatient prescription drug  coverage. Beneficiaries have until May 15, 2006, to select a plan without  the risk of penalties in the form of higher premiums.
Given the newness and complexity of the Part D benefit, it is critical that  beneficiaries and their advisers, including members of their families,  understand the available options so that beneficiaries can make informed  decisions on whether to enroll in Part D, and if so, which drug plan to  choose. Beneficiaries need to compare drug plans in light of their  anticipated prescription drug needs and existing arrangements for paying  for these drugs. In addition to comparing costs and drug coverage,  beneficiaries need to consider whether the plans they are comparing have  contracted with a local or mail-order pharmacy that will provide a  convenient means of filling their prescriptions.
As part of its responsibilities, CMS has undertaken outreach and education  efforts to provide beneficiaries and their advisers with the information  they need about Part D through various media, including written  documents, the 1-800-MEDICARE help line, and the Medicare Web site.  As of December 2005, CMS has produced more than 70 written documents  to explain Part D to beneficiaries. Medicare & You—the beneficiary  handbook—is the most widely available of these documents and was sent  directly to beneficiaries in October 2005. Other CMS documents are  targeted to specific groups of beneficiaries, such as dual-eligible  beneficiaries and beneficiaries with Medicare Advantage or Medigap  policies. Since March 1999, CMS has administered its nationwide   1-800-MEDICARE help line to answer beneficiaries’ questions about the  Medicare program. As of December 2005, about 7,500 customer service  representatives (CSR) were handling calls on the help line, which operates  24 hours a day, 7 days a week, and is run by two CMS contractors. Calls  are answered by an automated system and are routed to a CSR for specific  questions, including those about Part D. CMS provides CSRs with detailed  scripts to use in answering the questions. CSRs type in related keywords  to generate a list of suggested scripts for a given question, select the script  they consider best suited to the question, and read excerpts or the entire  script. Call center contractors write the scripts, and CMS checks them for  accuracy and completeness. CSRs can also consult other information  sources, such as the Medicare Web site. CMS does not allow CSRs to offer  individualized guidance to callers, including advice in choosing a drug  plan. CMS’s Medicare Web site provides information about all aspects of  the Medicare program. The Web site contains basic information about the  Part D benefit; suggests factors for beneficiaries to consider when  choosing a plan; describes situations common to beneficiaries with  guidance on next steps to take in deciding whether to enroll and what plan  to choose; lists frequently asked questions; and allows users to view, print,  or order publications. In addition, the site contains information on cost,  coverage, and convenience of individual plans. There is also a tool that  allows beneficiaries to enroll directly in the plan they have chosen.
CMS has also arranged for State Health Insurance Assistance Programs  (SHIP) to provide Part D information on request to Medicare beneficiaries  and their advisers. Currently, CMS provides grants to the 54 SHIPs—one in  each state, the District of Columbia, the Virgin Islands, Puerto Rico, and  Guam. State SHIPs provide subgrants to over 1,300 local organizations to  assist SHIPs in their efforts. In total, SHIPs rely on over 12,000 trained  counselors, most of whom are volunteers, to provide free counseling and  assistance via telephone and face-to-face sessions, public education  presentations and programs, and media activities.
Widespread confusion among beneficiaries about the costs and coverage  under the new benefit has been reported by the media and others. For  example, according to an October 2005 survey by a research organization,  some beneficiaries are unaware of the penalties for late enrollment and  others did not realize that beneficiaries had to sign up for the benefit. In  light of your interest in ensuring that Medicare beneficiaries receive the  information they need to make informed decisions, you asked us to  examine the quality of the information being provided on the Part D  benefit. In this report, we examined  the extent to which CMS’s written documents describe the Part D benefit  in a clear, complete, and accurate manner;  the effectiveness of CMS’s 1-800-MEDICARE help line in providing  accurate, complete, and prompt responses to callers inquiring about the  Part D benefit;    whether CMS’s Medicare Web site presents information on the Part D  benefit in a usable manner; and    how CMS has used SHIPs to respond to the needs of Medicare  beneficiaries for information on the Part D benefit.
We briefed your staff regarding the results of our review on   April 19, 2006. Appendix I contains information we provided during our  briefing to your staff.
To evaluate CMS’s written documents describing the Part D benefit, we  examined 70 relevant CMS publications and selected a sample of six  documents for in-depth review. These documents represent a variety of  document types, content, and target audiences and include Section 6 of  the Medicare & You beneficiary handbook, which discusses Part D. To  assess the clarity of the sample documents, we contracted with the  American Institutes for Research (AIR), a firm with experience in  evaluating written documents. AIR evaluated the texts by using three  standard readability tests; 60 commonly recognized good  communications practices; and user testing with 11 Medicare beneficiaries  and 5 advisers to beneficiaries, all of whom were asked to perform   18 specified tasks related to enrollment, coverage, costs, penalty, and  informational resources and provide feedback about their experiences. To  evaluate completeness, we reviewed the sample documents to determine if  they included sufficient information for the beneficiaries to identify   (1) their next steps in deciding whether to enroll and what plan to choose  and (2) important factors, such as penalty provisions, that could affect  their coverage decisions. To evaluate accuracy, we reviewed the sample  documents for consistency with MMA, regulations, and CMS guidance.
To assess the accuracy, completeness, and promptness of the help line  responses, we made 500 calls to 1-800-MEDICARE, posing one of five  questions about Part D in each call so that each question was asked   100 times. To develop the questions, we considered topics listed on the  Medicare Web site and topics addressed in scripts frequently accessed by  CSRs. To develop our criteria for evaluating the accuracy and  completeness of CSRs’ responses, we used three resources: (1) the  prescription drug finder tool on the Medicare Web site, (2) the 1-800- MEDICARE scripts, and (3) input from CMS officials. We also recorded  the length of each call, including wait times, and the time it took to be  connected to a CSR.
To assess whether the Medicare Web site presents information on the Part  D benefit in a usable manner, we contracted with the Nielsen Norman  Group (NN/g), a firm with expertise in Web design. NN/g conducted three  evaluations: (1) it calculated an overall usability score for the site— considering factors such as site navigation, customer support, and  presentation of online forms—to reflect the ease of finding necessary  information and performing various tasks; (2) it determined the usability  of 137 detailed aspects of the Web site, including aspects of Web design,  online tools, and writing style; and (3) it tested the ability of seven  participants (five beneficiaries and two advisers to beneficiaries) to  complete a total of 34 user tests to determine the ease of performing a  variety of Web-related tasks, such as browsing the site and determining  how to join a plan. We also reviewed the results of CMS’s analysis of its  Web site’s compliance with requirements that federal government Web  sites be accessible to people with disabilities.
Finally, to examine how CMS has used SHIPs to meet the information  needs of beneficiaries regarding Part D, we obtained information about  SHIPs, their funding, changes made in response to the new benefit, and the  impact of Part D on the demand for SHIP services. In addition, we  interviewed CMS officials who monitor SHIP activities as well as SHIP  coordinators in the five states with the largest populations of Medicare  beneficiaries—California, Florida, New York, Pennsylvania, and Texas.
We performed our work from November 2005 through May 2006 in  accordance with generally accepted government auditing standards. For  more information on our methodology, see appendix II.
subsections: section_title: Conclusions
paragraphs: Within the past 6 months, millions of Medicare beneficiaries have been  making important decisions about their prescription drug coverage and  have needed access to information about the new Part D benefit to make  appropriate choices. CMS faced a tremendous challenge in responding to  this need and, within short time frames, developed a range of outreach and  educational materials to inform beneficiaries and their advisers about   Part D. To disseminate these materials, CMS largely added information to  existing resources, including written documents, such as Medicare & You;  the 1-800-MEDICARE help line; the Medicare Web site; and support for  SHIPs. However, CMS has not ensured that its communications to  beneficiaries and their advisers are provided in a manner that is  consistently clear, complete, accurate, and usable. Six months have passed  since these materials were first made available to beneficiaries, and their  limitations could result in confusion among those seeking to make  coverage decisions. Although the initial enrollment period for Part D will  end on May 15, 2006, CMS will continue to play a pivotal role in providing  beneficiaries with information about the drug benefit during the year and  in subsequent enrollment periods. CMS has an opportunity to enhance its  communications on the Part D benefit. This would allow beneficiaries and  their advisers to be better prepared when deciding whether to enroll in the  benefit, and if enrolling, which drug plan to choose.
section_title: Recommendations for Executive Action
paragraphs: In order to improve the Part D benefit education and outreach materials  that CMS provides to Medicare beneficiaries, we are recommending that  the CMS Administrator take the following four actions:    Ensure that CMS’s written documents describe the Part D benefit in a  manner that is consistent with commonly recognized communications  guidelines and that is responsive to the intended audience’s needs.
Determine why CSRs frequently do not search for available drug plans if  the caller does not provide personal identifying information.
Monitor the accuracy and completeness of CSRs’ responses to callers’  inquiries and identify tools targeted to improve their performance in  responding to questions concerning the Part D benefit, such as additional  scripts and training.  Improve the usability of the Part D portion of the Medicare Web site by  refining Web-based tools, providing workable site navigation features and  links, and making Web-based forms easier to use and correct.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from CMS (see  app. III). CMS said that it did not believe our findings presented a  complete and accurate picture of its Part D communications activities.  CMS discussed several concerns regarding our findings on its written  documents and the 1-800-MEDICARE help line. However, CMS did not  disagree with our findings regarding the Medicare Web site or the role of SHIPs. CMS also said that it supports the goals of our recommendations and is already taking steps to implement them, such as continually  enhancing and refining its Web-based tools.
CMS discussed concerns regarding the completeness and accuracy of our  findings in terms of activities we did not examine, as well as those we did.  CMS stated that our findings were not complete because our report did  not examine all of the agency’s efforts to educate Medicare beneficiaries  and specifically mentioned that we did not examine the broad array of  communication tools it has made available, including the development of  its network of grassroots partners throughout the country. We recognize  that CMS has taken advantage of many vehicles to communicate with  beneficiaries and their advisers. However, we focused our work on the  four specific mechanisms that we believed would have the greatest impact  on beneficiaries—written materials, the 1-800-MEDICARE help line, the  Medicare Web site, and the SHIPs. In addition, CMS stated that our report  is based on information from January and February 2006, and that it has  undertaken a number of activities since then to address the problems we  identified. Although we appreciate CMS’s efforts to improve its Part D  communications to beneficiaries on an ongoing basis, we believe it is  unlikely that the problems we identified in this report could have been  corrected yet given their nature and scope.
CMS raised two concerns with our examination of a sample of written  materials. First, it criticized our use of readability tests to assess the clarity  of the six sample documents we reviewed. For example, CMS said that  common multisyllabic words would inappropriately inflate the reading  level. However, we found that reading levels remained high after adjusting  for 26 multisyllabic words a Medicare beneficiary would encounter, such  as Social Security Administration. CMS also pointed out that some experts  find such assessments to be misleading. Because we recognize that there  is some controversy surrounding the use of reading levels, we included  two additional assessments to supplement this readability analysis—the  assessment of design and organization of the sample documents based on  60 commonly recognized communications guidelines and an examination  of the usability of six sample documents, involving 11 beneficiaries and 5  advisers.
Second, CMS expressed concern about our examination of the usability of  the six sample documents. The participating beneficiaries and advisers  were called on to perform 18 specified tasks, after reading the selected  materials, including a section of the Medicare & You handbook. CMS  suggested that the task asking beneficiaries and advisers to calculate their  out-of-pocket drug costs was inappropriate because there are many other  tools that can be used to more effectively compare costs. We do not  disagree with CMS that there are a number of ways beneficiaries may  complete this calculation; however, we nonetheless believe that it is  important that beneficiaries be able to complete this task on the basis of  reading Medicare & You, which, as CMS points out, is widely disseminated  to beneficiaries, reaching all beneficiary households each year. In addition,  CMS noted that it was not able to examine our detailed methodology  regarding the clarity of written materials—including assessments  performed by one of our contractors concerning readability and document  design and organization. We plan to share this information with CMS, once  our report has become public.
Finally, CMS took issue with one aspect of our evaluation of the   1-800-MEDICARE help line. Specifically, CMS said the 41 percent accuracy  rate associated with one of the five questions we asked was misleading,  because, according to CMS, we failed to analyze 35 of the 100 responses.  However, we disagree. This question addressed which drug plan would  cost the least for a beneficiary with certain specified prescription drug  needs. We analyzed these 35 responses to this question and found the  responses to be inappropriate. The CSRs would not provide us with the  information we were seeking because we did not supply personal  identifying information, such as the beneficiary’s Medicare number or date  of birth. We considered such responses inappropriate because the CSRs  could have answered this question without personal identifying  information by using CMS’s Web-based prescription drug plan finder tool.  Although CMS said that it has emphasized to CSRs, through training and  broadcast messages, that it is permissible to provide the information we  requested without requiring information that would personally identify a  beneficiary, in these 35 instances, the CSR simply told us that our question  could not be answered. CMS also said that the bulk of these inappropriate  responses were related to our request that the CSR use only brand-name  drugs. This is incorrect—none of these 35 responses were considered  incorrect or inappropriate because of a request that the CSR use only  brand-name drugs—as that was not part of our question.
As arranged with your offices, unless you publicly announce the contents  of this report earlier, we plan no further distribution of it until 30 days  after its date. At that time, we will send copies of this report to the  Secretary of Health and Human Services, the Administrator of the Centers  for Medicare & Medicaid Services, and other interested parties. We will  also make copies available to others on request. In addition, the report will  be available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (312) 220-7600 or aronovitzl@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: Briefing on Medicare Part D
subsections: section_title: Purpose and Objectives
subsections: section_title: for overseeing this new benefit. CMS has taken steps to inform beneficiaries and their advisers about Part D using written documents, a toll-free help line, and an Internet Web site. CMS also gives State Health Insurance Assistance Programs (SHIP) funds to provide information about the Medicare program, including Part D.
section_title: Purpose and Objectives (continued)
subsections: section_title: Objective 1: Written Documents Methodology
subsections: section_title: sample of six CMS documents describing the Part D benefit. The sample was selected to represent a variety of document types, content, and target audiences.
paragraphs: Medicare & You (Section 6: Medicare  Prescription Drug Coverage)
section_title: Institutes for Research (AIR) to assess the clarity of sample documents.
paragraphs: Dual-eligible beneficiaries are Medicare beneficiaries  who receive full Medicaid benefits for services not  covered by Medicare.
Medicare Advantage replaces the Medicare+Choice  managed care program and expands the availability of  private health plan options to Medicare beneficiaries.
Do You Have a Medigap Policywith  Prescription Drug Coverage? sold by private insurers to help pay for Medicare cost- sharing requirements, as well as for some services not  provided by Medicare.
section_title: Objective 1: Written Documents Methodology (continued)
subsections: section_title: describing the Part D benefit, AIR
section_title: Objective 1: Written Documents Methodology (continued)
paragraphs: tested the usability of sample documents with 16 participants—11  Medicare beneficiaries, including 1 disabled beneficiary who was under 65, and 5 advisers to beneficiaries.
Everyone was asked to perform 18 specified tasks related to  enrollment, coverage, costs, penalty, and informational  resources. They were also asked to provide feedback about  their experiences.
Although the size of the group was small, research shows that  as few as 5 individuals can provide meaningful insights into  common problems.
section_title: Objective 1: Written Documents Methodology (continued)
subsections: section_title: To evaluate accuracy, we reviewed the sample documents for
paragraphs: consistency with laws, regulations, and CMS guidance.
section_title: Objective 1: Written Documents—Documents Lack Clarity
subsections: section_title: Readability assessment: Sample documents explaining the Part D
paragraphs: benefit are written at a reading level that is difficult for many seniors.
Reading levels for the sample documents were challenging for at  least the 40 percent of seniors, who read at or below the 5th grade  level.
Reading level estimates for the sample texts1 ranged from 7th grade to postcollege level.
Reading levels remain challenging for at least 40 percent of  seniors even after adjusting for 26 multisyllabic words, such as Medicare, Medicare Advantage, and Social Security  Administration. After the adjustment, the estimated reading level  ranged from 8th to 12th grade.
Estimates have a likely margin of error of ± two grades.
section_title: Objective 1: Written Documents—Clarity (continued)
subsections: section_title: Document design and organization assessment: The sample
paragraphs: documents demonstrated adherence to about half of the 60  commonly recognized written communications guidelines, on  average.
Desirable features: The documents  were written with a respectful and polite tone,  were free of clichés and slang,   contained useful contact information,   included concise and descriptive headings, and   generally followed graphic and formatting guidelines.
section_title: Objective 1: Written Documents—Clarity (continued)
paragraphs: Undesirable features: The documents  used too much technical jargon,  often did not define difficult terms,  included sentences and some paragraphs that were too long,   did not use sufficient summaries to assist the reader in  identifying key points.
section_title: Objective 1: Written Documents—Clarity (continued)
subsections: section_title: Usability assessment: Beneficiaries and advisers to beneficiaries
paragraphs: were frustrated by the documents’ lack of clarity and often could not  complete the 18 assigned tasks.
One of the 18 assigned tasks was completed by all beneficiaries  and advisers.
Eleven of the 18 assigned tasks were completed by at least half of  the beneficiaries and advisers.
Four of the 18 assigned tasks were completed by 2 or fewer of the  11 beneficiaries.
Nine of the 18 assigned tasks, were completed by 2 or fewer of  the 5 advisers.
section_title: Objective 1: Written Documents—Clarity (continued)
subsections: section_title: Some of the tasks that proved difficult included
paragraphs: computing projected total out-of-pocket costs for a plan that  provided Part D’s standard coverage (successfully completed by  none of the 11 beneficiaries and 2 of the 5 advisers),  evaluating whether it was possible to enroll in Medicare Part D  and keep drug coverage from a retiree health plan (successfully  completed by 2 beneficiaries and 2 advisers), and   determining the course of action for dual-eligibles who are  automatically enrolled in a plan that does not cover all drugs used  (successfully completed by 4 beneficiaries and 1 adviser).
section_title: Objective 1: Written Documents—Clarity (continued)
subsections: section_title: Participants described documents as too wordy, confusing, and hard
paragraphs: to follow.
Participants struggled with technical terms, such as “classes of commonly prescribed drugs” and “formulary,” which is a list of drugs  covered by a plan.
Even when most participants were able to complete the tasks, they  expressed confusion and frustration.
section_title: Objective 1: Written Documents—Documents Are Generally Complete
subsections: section_title: All documents reviewed provided sources of assistance and
paragraphs: relevant contact information, which could aid in identifying next  steps for coverage decisions.
All documents reviewed provided the dates of the start of initial  program enrollment and coverage.
section_title: Objective 1: Written Documents— Completeness (continued)
subsections: section_title: The documents did not provide sufficient information about the
paragraphs: cumulative effect of the penalty for missing the initial enrollment  deadline.
section_title: Objective 1: Written Documents—Documents Are Generally Accurate
subsections: section_title: Our analysis showed that the sample documents were generally
paragraphs: accurate and that the text was consistent with MMA, implementing regulations, and agency guidance.
However, we noted a few misleading statements in Medicare & You.
The document implied that if a beneficiary’s doctor applied for an  exception it would be granted, whereas exceptions to the  formulary are granted at each plan sponsor’s discretion.
The document outlined the minimum requirements for standard  coverage by Part D plans. However, it did not indicate that few  plans offer this exact coverage and that beneficiaries should be prepared to compare plans with varying premiums, co-payments,  and covered drugs to choose plans that best suit them.
section_title: Objective 2: 1-800-MEDICARE Help Line Methodology
subsections: section_title: different days of the week from January 17 to February 7, 2006, to match the daily and hourly pattern of calls reported by 1-800-MEDICARE in October 2005.
section_title: Objective 2: 1-800-MEDICARE Help Line Methodology (continued)
subsections: section_title: To evaluate the accuracy and completeness of CSRs’ responses to
paragraphs: our five questions, we used three resources:  the prescription drug finder tool on the Medicare Web site,   the 1-800-MEDICARE scripts prepared by CMS and contractors  for CSRs to use in responding to callers’ questions, and    input from CMS officials on the criteria we used to evaluate  responses.
To evaluate the promptness of the help line in answering calls, we  recorded the length of time it took to connect to a CSR for each call.
section_title: Objective 2: 1-800-MEDICARE Help Line Methodology (continued)
section_title: Objective 2: 1-800-MEDICARE Help Line Methodology (continued)
subsections: section_title: Question
section_title: 1. What drug plan can a beneficiary get that will cover all of his/her drugs at a pharmacy; have a mail-order option; and cost the least amount annually with [or without] a deductible?
paragraphs: An accurate and complete response  would identify the prescription drug plan  that has the lowest estimated annual cost  for the drugs the beneficiary uses.
2. Can a beneficiary who is in a  nursing home and not on  Medicaid sign up for a  prescription drug plan?
An accurate and complete response  would indicate that such a beneficiary can  choose whether to enroll in a Medicare  prescription drug plan.
section_title: Objective 2: 1-800-MEDICARE Help Line Methodology (continued)
subsections: section_title: Question
section_title: 3. Can a beneficiary enroll in the Medicare prescription drug program and keep his/her current Medigap policy?
paragraphs: An accurate and complete response would  inform the caller that enrolling for the prescription  drug benefit would depend on whether the  beneficiary’s Medigap plan was creditable—that  is, whether the coverage it provided was at least  as good as Medicare’s standard prescription  drug coverage—or noncreditable. The CSR  response would also mention that the  beneficiary’s Medigap plan should have sent  him/her information that outlines options.
section_title: Objective 2: 1-800-MEDICARE Help Line Methodology (continued)
subsections: section_title: Question
section_title: 4. What options does a beneficiary, who has retiree health insurance with prescription drug coverage that is not as good as the Medicare prescription drug coverage, have as it relates to the Medicare benefit?
paragraphs: An accurate and complete response  would indicate that a beneficiary has two  options: (1) keep current health plan and  join the prescription drug plan later with a  penalty; or (2) drop current coverage and  join a Medicare drug plan.
5. How do I know if a beneficiary  qualifies for extra help?
An accurate and complete response  would refer the beneficiary to the Social  Security Administration.
section_title: Objective 2: 1-800-MEDICARE Responses Often Accurate and Complete, but Some Not
subsections: section_title: We found that the 1-800-
section_title: MEDICARE help line provided accurate and complete answers to 334 of our 500 calls, a rate of about 67 percent. In addition, it provided accurate—but incomplete— answers for about 3 percent of our calls.
section_title: Objective 2: 1-800-MEDICARE—Variation in Results for Individual Questions
subsections: section_title: The accuracy and
paragraphs: completeness of responses  to our five questions varied  significantly, from 41 percent  to 90 percent.
section_title: Average for all
section_title: Objective 2: 1-800-MEDICARE—Variation (continued)
subsections: section_title: CSRs answered some questions better than others. For example:
paragraphs: CSRs accurately and completely answered question 5 (whether a  beneficiary qualifies for extra help), which had a specific script, 90  percent of the time.
CSRs accurately and completely answered question 2 (whether a  beneficiary in a nursing home, who was not on Medicaid, could  sign up for the drug benefit) 79 percent of the time—even though  there was no specific script for the question.
section_title: Objective 2: 1-800-MEDICARE—Variation (continued)
paragraphs: CSRs’ responses for question 3 (whether a beneficiary with a  Medigap policy could enroll in the drug benefit) were accurate and  complete 66 percent of the time. Many of the responses were  inaccurate because they did not provide adequate information  about creditable and noncreditable coverage.
The accuracy and completeness rate for question 4 (about retiree health insurance) was 58 percent. Many of the responses were  inaccurate because the CSRs did not follow the available script or  provide sufficient information about the implications of the  beneficiary's decision.
section_title: Objective 2: 1-800-MEDICARE—Variation (continued)
paragraphs: CSRs’ responses to question 1 (which requires CSRs to use the  prescription drug plan finder Web tool) were accurate and  complete less than 50 percent of the time. The rate is largely  caused by CSRs’ inappropriate responses—35 out of 100 times— that they were unable to answer the question without personal  identifying information, such as the beneficiary’s Medicare number  or date of birth.
section_title: Objective 2: 1-800-MEDICARE—Variation (continued)
subsections: section_title: Unintentional disconnections occurred when the system
paragraphs: inadvertently disconnected the call (19 calls).
Intentional disconnections were programmed by the telephone  company when wait times were projected to exceed 20 minutes  (3 calls).
The prescription drug plan finder Web tool used by CSRs was not  operative at the time of our call (1 call).
section_title: Objective 2: 1-800-MEDICARE—Variation in Wait Times
subsections: section_title: The amount of time we had to wait to speak with a CSR varied
paragraphs: significantly, ranging from no wait to more than 55 minutes.
About 75 percent of calls were connected in less than 5 minutes.
For calls where we waited more than 5 minutes to speak to a CSR, the wait time ranged from 5 minutes to over 55 minutes.
Sixty-two calls were on hold from 5 to 14 minutes, 59 seconds.  Thirty-nine calls were on hold from 15 to 24 minutes,  59 seconds.
Twenty-five calls were on hold 25 minutes or more.
For both intentional and unintentional disconnections, we often  waited more than 5 minutes before the disconnection occurred. In one case, we were placed on hold for 54 minutes before being  disconnected.
section_title: Objective 3: Medicare Web Site Methodology
subsections: section_title: Web site, we contracted with Nielsen Norman Group (NN/g), a firm with expertise in Web design.
section_title: Objective 3: Medicare Web Site Methodology (continued)
subsections: section_title: Section 508 requires that all federal Web sites be designed to
paragraphs: make information and services fully available to individuals with  disabilities.
Our review included an examination of CMS’s March 2006 report  assessing the compliance of its Medicare Web site with this  federal requirement and discussions with CMS officials.
section_title: Objective 3: Medicare Web Site Methodology (continued)
paragraphs: NN/g performed the following three separate evaluations:  Evaluation one: NN/g calculated an overall score of the site’s  usability, to reflect the ease of finding necessary information and  performing various tasks. For this calculation, NN/g considered  various factors, such as site navigation, customer support, and  presentation of online forms.
section_title: Objective 3: Medicare Web Site Methodology (continued)
paragraphs: Evaluation two: NN/g evaluated in detail the usability of 137  detailed aspects of the Part D benefit portion of the Web site.  Topics included   Web design (e.g., home page, navigation, search function,  graphics, and organization);  tools (e.g., plan finder);  writing style (e.g., tone, content, legibility, and readability);   accessibility (e.g., availability of site version for the blind); and   languages (e.g., links for users who have difficulty reading  English).
section_title: Objective 3: Medicare Web Site Methodology (continued)
paragraphs: Evaluation three: NN/g conducted a total of 34 user tests to  determine the ease of performing a variety of Web-related tasks,  such as browsing the site, making a change in address, finding  plan information under certain scenarios, comparing Medigap and  Part D drug coverage, and determining how to join a plan.
NN/g asked five Medicare beneficiaries—who were not  disabled—and two advisers to beneficiaries to perform one or  more user tests each using the Web site.
At the end of the user tests, the seven participants were asked  to provide feedback about their experiences.
section_title: Objective 3: Medicare Web Site Difficult to Use
subsections: section_title: for improvement. The usability score was 47 percent for seniors and 53 percent for younger adults. While there is no widely accepted benchmark for usability, these scores indicate that using the site can be difficult. For example, tools such as the drug plan finder were complicated to use, and forms that collect information online from users were difficult to correct if the user made an error.
section_title: Objective 3: Medicare Web Site Difficult to Use (continued)
section_title: Objective 3: Medicare Web Site Difficult to Use (continued)
paragraphs: information to assist navigation was often not helpful—for  example, text labels associated with links were not always  functioning; and  the writing style presented some challenges—for example,  material was written at the 11th grade level.
section_title: Objective 3: Medicare Web Site Difficult to Use (continued)
paragraphs: For evaluation three, the 34 user tests showed that the site was a  challenge for the seven participants to use. For example:  For 12 of the 34 tests, participants’ initial reactions were that  they would not be able to complete the tests and wanted to quit  trying.
On average, participants were able to proceed slightly more  than halfway through each of the 34 tests.
When asked for feedback on their experience with using the  site, the seven participants, on average, indicated high  frustration levels and low satisfaction.
section_title: Objective 3: Medicare Web Site Difficult to Use (continued)
subsections: section_title: established features to make information on its Medicare Web site accessible to disabled users. For example, CMS provides a “screen reader” version of the site for the visually impaired. This technology translates text and data into spoken words.
section_title: Objective 3: Medicare Web Site Difficult to Use (continued)
subsections: section_title: CMS’s March 2006 review of its site’s compliance with section 508
paragraphs: showed that two requirements were not met:   The plan finder did not provide alternative text for all images—that  is, there was no text for the screen reader to read. Therefore,  images could not be translated into spoken words for the visually  impaired.
The plan finder did not allow screen readers to recognize form  fields and translate forms into spoken words. As a result, visually  impaired users would not have been able to complete Web-based  forms.
A CMS official told us that the agency made the necessary  corrections on April 20, 2006, but we did not verify that these  corrections were made.
section_title: Objective 4: SHIP Methodology
subsections: section_title: Objective 4: SHIPs’ Responses to Beneficiaries’ Needs Concerning Part D
subsections: section_title: the Part D benefit, CMS increased SHIP funding in recent years. Funding for the 2003 SHIP grant year2 was $12 million, and it reached $31.7 million for the 2005 grant year. CMS kept funding relatively high for the 2006 grant year—$30 million—to ensure that SHIPs continue to play an important role in educating beneficiaries about the Part D benefit.
paragraphs: A SHIP grant year begins on April 1 of the year the funds become available.
section_title: Objective 4: SHIPs’ Responses to Part D (continued)
subsections: section_title: million people. According to preliminary data for the first 9 months of the 2005 SHIP grant year—when CMS was gearing up its outreach and education on Part D—SHIPs served approximately 3.3 million individuals, an increase of nearly 770,000 from the prior full grant year. CMS attributes this increase in demand for services to beneficiaries’ need for assistance on the Part D benefit.
section_title: Objective 4: SHIPs’ Responses to Part D (continued)
subsections: section_title: The average number of calls referred from the 1-800-MEDICARE
paragraphs: help line to SHIPs has increased significantly.
The monthly average of number of calls referred to SHIPs  increased from 16,000 referrals for May through September 2005  to approximately 43,000 for October and November 2005, the  months around the time when enrollment in the Part D benefit  began.
According to CMS officials, this increased demand was influenced by callers seeking advice on choosing a drug plan. Unlike CSRs  on the help line, SHIP counselors can offer individualized  guidance to callers.
section_title: Objective 4: SHIPs’ Responses to Part D (continued)
subsections: section_title: California served over 120,000 people in January 2006, compared
paragraphs: to about 35,000 served in all of 2005.
Florida, mostly during November and December of 2005, held at  least six “phone bank” events—where SHIP counselors were  available to take calls on the Part D benefit during live newscasts.  Florida plans to hold two additional phone banks as the May 15  enrollment deadline approaches.
section_title: Objective 4: SHIPs’ Responses to Part D (continued)
paragraphs: New York reported nearly doubling its formal training sessions for  SHIP counselors in 2005, to prepare them for the demand for  services related to the Part D benefit.
Texas counseled 45,719 clients and conducted 523 outreach  events from November 15, 2005—the official start of the  enrollment period—to March 22, 2006.
Pennsylvania held over 3,000 enrollment events, which were  attended by more than 130,000 people, from May 2005 to  February 28, 2006.
section_title: Objective 4: SHIPs’ Responses to Part D (continued)
subsections: section_title: Since December 2005, CMS has been conducting biweekly
paragraphs: meetings with its regional offices, which interact directly with SHIP  offices, to gauge SHIPs’ ability to meet the demands of beneficiaries.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: In this report, we assessed (1) the extent to which the Centers for  Medicare & Medicaid Services’ (CMS) written documents describe the  Medicare Part D prescription drug benefit in a clear, complete, and  accurate manner; (2) the effectiveness of CMS’s 1-800-MEDICARE help  line in providing accurate, complete, and prompt responses to callers  inquiring about the Part D benefit; (3) whether CMS’s Medicare Web site  presents information on the Part D benefit in a usable manner; and   (4) how CMS has used State Health Insurance Assistance Programs (SHIP)  to respond to the needs of Medicare beneficiaries for information on the  Part D benefit. To obtain information on CMS’s efforts to educate  beneficiaries about Part D, we interviewed agency officials responsible for  Part D written documents, the 1-800-MEDICARE help line, the Medicare  Web site, and SHIPs. Following our briefing of congressional staff on April  19, 2006, the briefing slides were updated to reflect CMS’s reported  correction to the Medicare Web site to comply with section 508 of the  Rehabilitation Act of 1973. We determined that the data used were  sufficiently reliable for the purposes of this report.
subsections: section_title: Written Documents
paragraphs: To assess the clarity, completeness, and accuracy of written documents,  we compiled a list of all available CMS-issued Part D benefit publications  intended to inform beneficiaries and their advisers and selected a sample  of 6 from the 70 CMS documents available, as of December 7, 2005, for in- depth review, as shown in table 1. The sample Part D documents were  chosen to represent a variety of publication types, such as frequently  asked questions and fact sheets available to beneficiaries about the Part D  drug benefit. We selected documents that targeted all beneficiaries or  those with unique drug coverage concerns, such as dual-eligibles and  beneficiaries with Medigap.
To evaluate clarity, we contracted with the American Institutes for  Research (AIR)—a firm with experience in evaluating written material.  AIR evaluated the texts of the six sample documents using three  methodologies:  1.  three standard readability tests;  2.  60 commonly recognized written communications guidelines, including  practices to aid senior readers; and  3.  user testing with 11 Medicare beneficiaries and 5 advisers to  beneficiaries, who performed 18 specified tasks related to enrollment,  coverage, cost, penalty, and information resources and provided  feedback about their experiences.
We reviewed the sample documents for completeness to determine  whether they contained sufficient information to allow the beneficiaries to  identify (1) their next steps in determining whether to enroll and what  plan to choose and (2) important factors, such as penalty provisions, that  could affect their coverage decisions. To identify those important factors  associated with the Part D benefit, we reviewed relevant laws, regulations,  and 1-800-MEDICARE scripts prepared for customer service  representatives (CSR) to read to callers and obtained information from  advocacy groups. To evaluate the accuracy of information, we reviewed  the sample materials for compliance with laws, regulations, and CMS  guidance.
section_title: The 1-800-MEDICARE Help Line
paragraphs: To determine the accuracy and completeness of information provided  regarding the Part D benefit, we placed a total of 500 calls to the 1-800- MEDICARE help line. We posed one of five questions about Part D in each  call, so that each question was asked 100 times. Each question was  pretested before we finalized its wording. We randomly placed calls at  different times of the day and different days of the week from January 17  to February 7, 2006. Our calling times were chosen to match the daily and  hourly pattern of calls reported by 1-800-MEDICARE in October 2005. We  informed CMS officials that we would be placing calls; however, we did  not tell them the questions we would ask or the specific dates and times  that we would be placing our calls.
To select the five questions, we considered topics identified in the  Medicare Web site’s frequently asked questions. In addition, we  considered topics most frequently addressed by 1-800-MEDICARE CSRs  based on help line reports. To evaluate the accuracy of CSRs’ responses to  our five questions, we used three resources: (1) the prescription drug plan  finder tool on the Medicare Web site, (2) 1-800-MEDICARE scripts, and   (3) input obtained from CMS officials on the criteria we used for  evaluating CSR responses. Table 2 lists the questions we asked and the  criteria we used to evaluate the accuracy of responses.
When placing our calls, we identified ourselves as a beneficiary’s relative,  but did not provide CSRs with specific identifying information, such as a  Medicare beneficiary number or date of birth. During our calls, CSRs were  not aware that their responses would be included in a research study. We  recorded the length of each call, including wait times, and the time it took  before being connected to a CSR. We evaluated the accuracy and  completeness of the responses by CSRs to the 500 calls by determining  whether key information was provided.
The results from our 500 calls are limited to those calls and are not  generalizable to the universe of calls made to the help line. The questions  we asked were limited to matters concerning the Part D benefit and do not  encompass all of the questions callers might ask.
section_title: Medicare Web Site
paragraphs: We contracted with the Nielsen Norman Group (NN/g)—a firm with  expertise in Web design—to assess the usability of the Part D information  available on the Medicare Web site. This study consisted of three separate  evaluations. First, NN/g compared the site’s compliance with established  usability guidelines to determine a usability score to reflect the ease of  finding necessary information and performing various tasks. Specifically,  to determine the usability scores, NN/g evaluated various aspects of the  Web site using industry-recognized “good” Web design practices, as  indicated by the contractor, and the collective body of knowledge from  NN/g internal reports and experts, or NN/g usability guidelines.
Second, NN/g determined the degree of difficulty associated with 137  detailed aspects of Web site design for the Part D portion of the site. The  137 aspects fall into the following general categories:    overall Web design (e.g., home page, navigation, search function, graphics,  and overall organization);  tools (e.g., plan finder);    writing style (e.g., content, tone, legibility, and readability);    accessibility (e.g., availability of a version of the Web site for the blind);  and  languages (e.g., availability of languages other than English).
NN/g determined the difficulty level in using each of the 137 aspects. NN/g  noted aspects that had good design and would not be expected to cause  confusion. For those aspects with a design that would be expected to  cause confusion, NN/g ranked the associated difficulty level as high,  medium, or low.
Third, NN/g performed a qualitative evaluation on January 20 and 23, 2006,  to test the ability of five Medicare beneficiaries and two beneficiary  advisers to perform specified tasks related to Medicare beneficiaries using  the Web site and to obtain feedback about participants’ experiences. While  the results are not statistically valid, these users provided important  insights into the usability of the Medicare Web site. Participants were  asked to “think out loud” as they worked through their tasks, while an  NN/g facilitator observed their behavior and took notes. NN/g gave each  task a score. At the end of their sessions, NN/g asked participants for input  regarding their confidence in the answers they obtained from the Web site,  and their overall satisfaction and frustration levels associated with using  the site.
Finally, we obtained the results of CMS’s March 2006 review of its Web  site’s compliance with section 508 of the Rehabilitation Act of 1973, as  amended. This law requires federal agencies to make the information on  their Web sites accessible to people with disabilities. We also discussed  the results of this review with agency officials and followed up with them  to determine the status of CMS’s corrective actions.
section_title: State Health Insurance Assistance Programs
paragraphs: To determine the role of SHIPs in helping Medicare beneficiaries  understand Part D, we interviewed CMS officials who monitor SHIPs’  activities. We also reviewed information that we obtained from CMS  officials and other sources on the program, its funding, changes made in  response to the introduction of Part D, and the impact of Part D on the  demand for SHIP services. In addition, we interviewed SHIP officials in  California, Florida, New York, Texas, and Pennsylvania—the five states  with the largest Medicare populations—to obtain information on the  experience of their SHIPs with Part D.
We conducted our work from November 2005 through May 2006 in  accordance with generally accepted government auditing standards.
section_title: Appendix III: Comments from the Centers for Medicare & Medicaid Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Susan T. Anthony and Geraldine  Redican-Bigott, Assistant Directors; Ramsey L. Asaly; Enchelle Bolden;  Laura Brogan; Shaunessye D. Curry; Chir-Jen Huang; M. Peter Juang;   Ba Lin; Michaela M. Monaghan; Roseanne Price; Pauline Seretakis;   Margaret J. Weber; and Craig H. Winslow made contributions to this  report.