Report Template Version = 01-13-05 

Department of Health and Human Services


OFFICE OF 

INSPECTOR GENERAL 

MEDICARE BENEFICIARY 


TELEPHONE CUSTOMER 


SERVICE


Daniel R. Levinson


Inspector General 


August 2005


OEI-07-04-00030

Report Template Version = 01-13-05 

Office of Inspector General 
 http://oig.hhs.gov 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, 

as amended, is to protect the integrity of the Department of Health and Human Services 

(HHS) programs, as well as the health and welfare of beneficiaries served by those 

programs. This statutory mission is carried out through a nationwide network of audits, 

investigations, and inspections conducted by the following operating components: 

Office of Audit Services 

The OIG's Office of Audit Services (OAS) provides all auditing services for HHS, either by 

conducting audits with its own audit resources or by overseeing audit work done by others. 

Audits examine the performance of HHS programs and/or its grantees and contractors 

in carrying out their respective responsibilities and are intended to provide independent 

assessments of HHS programs and operations in order to reduce waste, abuse, and 

mismanagement and to promote economy and efficiency throughout the department. 

Office of Evaluation and Inspections 

The OIG's Office of Evaluation and Inspections (OEI) conducts management and 

program evaluations (called inspections) that focus on issues of concern to the 

department, the Congress, and the public. The findings and recommendations contained 

in the inspections reports generate rapid, accurate, and up-to-date information on the 

efficiency, vulnerability, and effectiveness of departmental programs. The OEI also 

oversees State Medicaid fraud control units, which investigate and prosecute fraud and 

patient abuse in the Medicaid program. 

Office of Investigations 

The OIG's Office of Investigations (OI) conducts criminal, civil, and administrative 

investigations of allegations of wrongdoing in HHS programs or to HHS beneficiaries 

and of unjust enrichment by providers. The investigative efforts of OI lead to criminal 

convictions, administrative sanctions, or civil monetary penalties. 

Office of Counsel to the Inspector General 

The Office of Counsel to the Inspector General (OCIG) provides general legal services to 

OIG, rendering advice and opinions on HHS programs and operations and providing all 

legal support in OIG's internal operations. The OCIG imposes program exclusions and 

civil monetary penalties on health care providers and litigates those actions within the 

department. The OCIG also represents OIG in the global settlement of cases arising 

under the Civil False Claims Act, develops and monitors corporate integrity agreements, 

develops compliance program guidances, renders advisory opinions on OIG sanctions to 

the health care community, and issues fraud alerts and other industry guidance. 
Report Template Version = 01-13-05 

Δ EXECUTIVE SUMMAR Y 

OBJECTIVE 

To assess callers’ experiences with Medicare-funded call centers and to 

determine priorities for and efforts to ensure quality customer service. 

BACKGROUND 

The Centers for Medicare & Medicaid Services (CMS) serves the 

informational needs of approximately 40 million Medicare beneficiaries 

and/or their representatives by using multiple communication sources. 

The most heavily used communication sources are the Medicare-funded 

call centers. At the time of our fieldwork, beneficiaries could call either 

a fee-for-service contractor call center or 1-800-MEDICARE with 

questions about Medicare. There were 29.3 million calls to 

Medicare-funded call centers in fiscal year 2004. 

We administered a survey to 305 callers over a 1-week period to assess 

their experiences with Medicare telephone customer service. We asked 

callers if they were satisfied with the customer service they received, if 

they believed their questions were answered, and if they received all the 

information they needed. We also asked callers about their priorities 

for customer service. We administered a survey to the managers of all 

the call centers to determine the call center managers’ priorities for 

quality customer service and to identify any quality improvement efforts 

planned or underway. Finally, we reviewed CMS’s oversight activities 

related to call centers, including accompanying CMS staff on a 

performance evaluation of a large fee-for-service call center. 

FINDINGS 

Eighty-four percent of callers asked to rate satisfaction were 

satisfied overall with the customer service they received; however, 

44 percent of all callers reported difficulty accessing information 

from call centers. The 44 percent of callers who had difficulty 

accessing information reported at least one of the following experiences: 

(1) finding the Interactive Voice Response not easy to use, (2) not 

receiving an answer to their question or all the information they 

needed, or (3) not receiving the answer to their question as quickly as 

desired. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE i 
Report Template Version = 01-13-05 

EXE CUTIVE S U MMAR Y

Callers and call center managers both placed a high priority on 

accuracy of answers, yet evidence suggests oversight of accuracy 

may be inadequate. Sixty-seven percent of all callers and 71 percent of 

call center managers ranked accuracy as their highest priority. 

Although we did not test accuracy of answers specifically, we believe it 

is important to note that 24 percent of callers reported not receiving an 

answer to their question or all the information they needed. Both of 

these experiences are related to accuracy. Findings in two Government 

Accountability Office (GAO) reviews of Medicare-funded call centers 

raised concerns that oversight of accuracy may be inadequate. A 

July 2004 GAO report found that the performance evaluation criteria 

“. . . are not designed to verify that [customer service representatives’] 

responses to providers are accurate.” A December 2004 GAO report 

found that “CMS and its contractors do not emphasize [customer service 

representatives’] ability to answer questions accurately using [CMSapproved guidance].” While limited, our fieldwork supported GAO’s 

findings. The CMS performance evaluation we observed focused on 

validation of reported performance results (e.g., number of monitored 

calls, number of calls answered in less than 60 seconds, average speed of 

answer) and documented training activities. However, the review team 

spent little time assessing the accuracy of answers given to callers. 

Some call centers conducted quality assurance activities that focused on 

accuracy, exceeding CMS requirements. These activities included 

increasing the number of monitored calls in the Quality Call Monitoring 

process, tying monitored performance to customer service 

representative evaluations, involving representatives in the 

administration of quality assurance activities, and administering 

periodic knowledge tests. 

RECOMMENDATIONS 

We recommend that CMS: 

o Strengthen current oversight to place greater emphasis on 

completeness of responses, greater efficiency for the caller, and 

accuracy of answers given by customer service representatives. 

CMS could achieve this by: (1) including CMS staff with expertise 

in call handling on national review teams for performance 

evaluations, (2) requiring call centers to conduct periodic 

knowledge testing with minimum passing scores, and/or 

(3) increasing the number of calls monitored in the Quality Call 

OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE ii 
Report Template Version = 01-13-05 

EXE CUTIVE S U MMAR Y 

Monitoring process. Given that some call centers are currently 

engaged in conducting periodic knowledge testing and increased 

monitoring, CMS may want to determine the impact these 

practices may have on quality outcomes. 

o Continue to seek ways to improve the national Interactive Voice 

Response system. We note that subsequent to our fieldwork, CMS 

moved the option to speak with a customer service representative 

to the main menu, thereby decreasing the amount of time it 

required to arrive at this option from approximately 2 minutes to 

50 seconds. 

AGENCY COMMENTS 

In its comments to the draft report, CMS expressed appreciation for 

the Office of Inspector General’s analysis and recommendations to 

help improve the benficiary inquiry call centers. CMS stated it will 

continue with quality assurance activities, some of which we 

suggested in our recommendations, already underway at the 

1-800-MEDICARE call center. CMS described these activities in 

detail. However, CMS does not plan to invest significant resources 

into reengineering quality assurance activities at the fee-for-service 

call centers because this workload will be integrated into the 

1-800-MEDICARE operation. In addition, CMS stated that an expert 

in the field of interactive voice response is being consulted to assist 

with improving and enhancing the system. For CMS’s complete 

comments, see page 17 of this report. 

OFFICE OF INSPECTOR GENERAL RESPONSE 

We appreciate CMS’s comments to this report and note that CMS is 

taking action to address issues raised in this report. Changes were 

made to the report to reflect technical comments received from CMS. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE iii 
Report Template Version = 01-13-05 

Δ T ABLE OF CONTENTS 

EXECUTI VE SUMMARY ........................................ i

INTRODUCTI ON ................................................ 1 

FINDI NGS ....................................................... 8 

Positive overall caller satisfaction .......................... 8

Difficulty accessing information............................ 9

Priorities and quality assurance activities................... 12

RECOM M E NDATI O NS ......................................... 15 

AGENCY COMME NTS ......................................... 17

APPENDI XES .................................................. 24 

A: Confidence Intervals for Key Estimates and Chi-Square

 Tests ............................................. 24

B: Profile of Surveyed Callers ............................ 26

ACKN O W LEDGMENTS ......................................... 27
Report Template Version = 01-13-05 

Δ INTRODU C TION 

OBJECTIVE 

To assess callers’ experiences with Medicare-funded call centers and to 

determine priorities for and efforts to ensure quality customer service. 

BACKGROUND 

Beneficiary Telephone Inquiries 

The Centers for Medicare & Medicaid Services (CMS) serves the 

informational needs of approximately 40 million Medicare beneficiaries 

and/or their representatives by using multiple communication sources. 

The most heavily used communication sources are the Medicare-funded 

call centers. At the time of our fieldwork, beneficiaries could call either 

a fee-for-service contractor call center or 1-800-MEDICARE (which is 

also a CMS contractor) with questions about Medicare. Fee-for-service 

contractors (carriers, fiscal intermediaries, and durable medical 

equipment regional carriers) operated 63 separate call centers.1 In 

fiscal year (FY) 2004, the fee-for-service call centers handled 

approximately 12.8 million calls at a cost of $96.3 million, while 

1-800-MEDICARE handled 16.5 million calls at a cost of $104.2 million. 

Since Medicare’s inception in 1965, Medicare contractors have operated 

the telephone customer service system as a group of stand-alone call 

centers. The Balanced Budget Act of 1997 required a toll-free number 

for inquiries regarding the newly created Medicare+Choice program. In 

1998, CMS established 1-800-MEDICARE for this purpose. In 1999, the 

phone number was phased in as part of the National Medicare 

Education Program and offered general information about Medicare, 

health plan options, supplemental insurance, and referral telephone 

numbers for help with claims or more complex issues. The original 

purpose of 1-800-MEDICARE, a source of information about 

Medicare+Choice options, broadened as new initiatives began. 

Section 923(d) of the Medicare Prescription Drug, Improvement, and 

Modernization Act of 2003 (MMA) officially established 1-800-

MEDICARE as the primary source of general Medicare information and 

assistance. In July 2004, 1-800-MEDICARE became the single point of 

entry for the telephone customer service system, with transfer 

capabilities to the various fee-for-service call centers. 

1 CMS provided a listing of 63 fee-for-service call centers dated April 2, 2004. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 1 
Report Template Version = 01-13-05 

INTROD UCT ION 

CMS Goals for Customer Service Satisfaction 

According to the CMS Web site, CMS’s customer service mission is to 

continuously improve Medicare beneficiary customer satisfaction 

through the delivery of high quality and cost effective customer service. 

CMS’s FY 2004 Government Performance and Results Act Annual 

Performance Plan indicated that the performance goal for telephone 

customer service was to improve beneficiary telephone customer service 

in terms of accessibility, accuracy of response, and caller satisfaction. 

During FYs 2000-2002, CMS developed baselines and collected data 

related to this performance goal. However, in the plan narrative 

discussing the performance results, CMS indicated that in 

FYs 2003-2004, it redirected efforts and resources away from the 

performance goal of improving beneficiary telephone customer service in 

terms of accessibility, accuracy of response, and caller satisfaction 

(including funding for a national caller satisfaction survey), instead 

focusing on creating a single point of entry (1-800-MEDICARE). Thus, 

no current national baseline for beneficiary satisfaction exists. 

In FY 2004, only 3 of the 63 fee-for-service call centers conducted 

beneficiary satisfaction surveys, which were part of a performance 

incentive pilot program. At the time of our fieldwork, CMS required its 

1-800-MEDICARE contractor to conduct 400 beneficiary satisfaction 

surveys per month. 

CMS Requirements for Call Centers 

The Medicare Contractor Beneficiary and Provider Communications 

Manual outlines the guidelines for telephone customer service, 

including: hours of operation, bilingual services, services for people 

with hearing impairments, and requirements related to Interactive 

Voice Response (IVR).2,3 Certain performance metrics must be met in 

terms of how quickly calls are answered, how quickly callers who choose 

to speak with a customer service representative actually speak with a 

representative, how often callers’ questions are answered on the initial 

call, and how often callers receive a busy signal. 

2 An IVR is a software application in which someone uses a touch-tone telephone to interact 

with a database to acquire information. An IVR provides prerecorded voice responses 

based on voice or keypad input. 

3 Medicare Contractor Beneficiary and Provider Communications Manual, Chapter 2, 

section 20.1. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 2 
Report Template Version = 01-13-05 

INTROD UCT ION 

CMS requires all contractors to use Quality Call Monitoring (QCM) to 

monitor, measure, and report the quality of service they provide.4 The 

QCM is the primary method CMS uses to assess whether call centers 

are meeting established performance standards. 

The QCM process requires a quality auditor to monitor a minimum of 

three calls per customer service representative per month at 

fee-for-service call centers, and four calls per representative per month 

at 1-800-MEDICARE. The quality auditor uses a scorecard to rate the 

representative in areas such as adherence to the Privacy Act, customer 

skills (e.g., greeting, tone, volume), and knowledge skills (e.g., accuracy, 

completeness, call action). The following performance standards must 

be met for the calls monitored each month through the QCM: 

o The percentage of customer service representatives scoring as 

“Pass” for adherence to the Privacy Act should be no less than 

90 percent. 

o The percentage of customer service representatives scoring as 

“Achieves Expectation” or higher for customer skills should be no 

less than 90 percent. 

o The percentage of customer service representatives scoring as 

“Achieves Expectation” or higher for knowledge skills should be no 

less than 90 percent. 

The results of the scorecards are used for coaching the representatives. 

Scorecard results are aggregated and reported to CMS on a monthly 

basis. 

CMS Evaluation of Call Centers 

CMS validates the reported scorecard data during a fee-for-service 

contractor’s performance evaluation. According to the FY 2004 review 

protocol, the purpose of the performance evaluation is to determine 

whether the contractor is: (1) answering calls from Medicare 

beneficiaries in an efficient, accurate, and professional manner that 

meets established performance standards; (2) reporting performance 

data accurately to CMS; and (3) following established guidelines for 

training and coaching customer service representatives to properly 

respond to telephone inquiries. Based on FYs 2001-2003 contractor 

performance evaluation reports and the schedule of FY 2004 planned 

4 Medicare Contractor Beneficiary and Provider Communications Manual, Chapter 2, 

section 20.1.7. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 3 
Report Template Version = 01-13-05 

INTROD UCT ION 

evaluations that CMS provided to us, we determined that CMS 

conducted 31 performance evaluations in FYs 2001-2004 at 

23 fee-for-service call centers. 

CMS employs different evaluation methods for 1-800-MEDICARE. 

Every 4 months, the 1-800-MEDICARE contractor provides a narrative 

self-evaluation that CMS uses to conduct an assessment of: 

(1) performance and progress achieved in various functional areas 

(e.g., call center operations, information technology, response to 

requests for publications); (2) a comparison of actual performance to the 

standard for various performance metrics; and (3) a rating for contract 

compliance. In addition, a quality assurance contractor monitors the 

operation of the IVR and calls answered by customer service 

representatives on a monthly basis. 

Other Studies 

In a February 2002 study for which CMS contracted with a private 

consultant, beneficiaries who had contacted any one of the eight largest 

Medicare contractors with a billing question that was transferred to 

their respective fraud units were surveyed. Seventy-five percent of 

beneficiaries reported being satisfied with their overall experience. In a 

February 2002 report regarding provider inquiries entitled 

“Communications with Physicians can be Improved,” (GAO-02-249), the 

Government Accountability Office (GAO) found that the customer 

service representatives they tested provided complete and accurate 

answers to their questions 15 percent of the time. In a July 2004 

followup report entitled “Call Centers Need to Improve Responses to 

Policy-Oriented Questions from Providers,” (GAO-04-669), GAO found 

that only 4 percent of answers given were complete and accurate. 

Section 923 of the MMA mandated GAO to conduct a study of the 

accuracy and consistency of information provided through 

1-800-MEDICARE. This study, “Accuracy of Responses from the 

1-800-MEDICARE Help Line Should Be Improved,” was released in 

December 2004 (GAO-05-130). GAO reviewers called 

1-800-MEDICARE and posed test questions to the representatives 

regarding the Medicare program. GAO found that the representatives 

provided accurate answers 61 percent of the time. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 4 
Report Template Version = 01-13-05 

INTROD UCT ION

METHODOLOGY 

We used three methods to gather information for this inspection: 

(1) telephone surveys with callers to fee-for-service or 

1-800-MEDICARE call centers, (2) a review of CMS oversight activities, 

and (3) interviews with call center managers. 

Telephone Surveys 

We divided the call centers into three groups: large call centers, small 

call centers, and 1-800-MEDICARE. Large and small designations were 

determined by calculating the mean FY 2003 call volume per 

fee-for-service call center (223,609 calls). Fee-for-service call centers 

with FY 2003 call volumes below the mean were designated as small, 

and call centers with FY 2003 call volumes above the mean were 

designated as large. The FY 2003 call volume range for small call 

centers was 4,531 to 223,396 calls and the FY 2003 call volume range 

for large call centers was 241,066 to 1,118,851 calls. We obtained a 

telephone network generated listing of all calls made to small and large 

call centers and 1-800-MEDICARE call centers during the week of 

April 12-16, 2004.5 The listing included the following information: call 

center telephone number dialed, originating telephone number, and 

date and time of call. We spoke with CMS officials to ensure that the 

selected week was typical of weeks throughout the year. 

For each of 5 days (Monday through Friday), we eliminated duplicate 

originating telephone numbers (some callers called more than once in a 

given day) and randomly selected 75 callers. For each day, these callers 

were divided into 3 groups: 25 who called small call centers, 25 who 

called large call centers, and 25 who called 1-800-MEDICARE 

(75 x 5 days = 375 randomly selected callers) for a total of 15 strata. As 

illustrated in Table 1 on the next page, we completed 305 telephone 

surveys from the sample of 375 callers, for an 81 percent response rate. 

Ninety-seven percent of our surveys were completed within 2 business 

days of the caller’s sampled call to the call centers. The other 3 percent 

were completed within 5 business days. We administered our survey to 

determine callers’ experiences, such as whether callers believed their 

questions were answered, whether callers received all the information 

they needed, and how they rated their experience based on selected 

5 Small and large call centers operate Monday through Friday during typical business 

hours; 1-800-MEDICARE operates 7 days a week, 24 hours per day. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 5 
Report Template Version = 01-13-05 

INTROD UCT ION 

QCM performance standards. We did not ask 34 callers all survey 

questions because they ended their calls when they were unable to 

receive a response to their question from a customer service 

representative or IVR (e.g., called after business hours); therefore, 

271 callers were asked to rate their experiences related to the responses 

they received from the call center. 

Table 1: Population, Sampled Calls, and Completed Surveys by Strata 

Day of Week Type of Call Center Population Sampled Calls Completed Surveys 

Monday 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

15,309 

47,450 

32,716 

25 

25 

25 

22 

19 

20 

Tuesday 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

12,544 

40,277 

29,262 

25 

25 

25 

22 

22 

21 

Wednesday 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

11,550 

35,963 

27,811 

25 

25 

25 

20 

22 

19 

Thursday 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

11,113 

36,487 

29,852 

25 

25 

25 

15 

22 

21 

Friday 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

10,769 

35,131 

31,029 

25 

25 

25 

21 

20 

19 

Total 407,263 375 305 

Source: Office of Inspector General analysis of telephone network data, 2004. 

All 305 callers were asked questions about their priorities for customer 

service and their experiences during their calls. We performed 

comparisons with regard to overall customer service satisfaction, 

finding the IVR easy to use, receiving an answer to the caller’s question 

and all of the information needed, and receiving an answer to the 

caller’s question as quickly as desired. We analyzed results with respect 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 6 
Report Template Version = 01-13-05 

INTROD UCT ION

to the three groups: callers to small call centers, callers to large call 

centers, and callers to 1-800-MEDICARE. Findings are projected to the 

week of April 12-16, 2004. 

We did not administer test questions to customer service 

representatives to directly test the accuracy of answers given to callers. 

However, we did ask callers if they received an answer to their question 

and whether they received all of the information they needed. We chose 

not to directly test accuracy due to the December 2004 GAO report 

addressing this issue. To gain firsthand experience with the 

1-800-MEDICARE IVR, we called it on five separate occasions in 

August 2004. 

Review of CMS Oversight Activities 

We obtained the date and results of all call center performance 

evaluations conducted in FY 2001, FY 2002, and FY 2003, as well as 

improvement plans, action steps, progress, and followup of the results. 

We accompanied CMS staff on a performance evaluation of a large call 

center in June 2004. We observed the sampling methods, review 

procedures, and protocols used in conducting the performance 

evaluation. 

Structured Interviews with Call Center Managers 

We conducted a total of 55 telephone interviews with managers from all 

of the small and large call centers and 1-800-MEDICARE call centers to 

determine the call center managers’ priorities for quality customer 

service and to identify any quality improvement efforts planned or 

underway.6 We asked for results data and opinions regarding these 

efforts. 

Standards 

This inspection was conducted in accordance with the Quality 

Standards for Inspections issued by the President’s Council on 

Integrity and Efficiency. 

6 Some large fee-for-service contractors operated multiple call centers. We combined some 

interviews, meaning that the total number of interviews conducted is less than the total 

number of call centers. 

OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 7 
Report Template Version = 01-13-05 

Δ FINDINGS

Of the 305 callers with whom we 

Eighty-four percent of callers asked to rate spoke, 271 received responses to 

satisfaction were satisfied overall with the their questions from customer 

customer service they received service representatives or IVRs. 

Eighty-four percent of these callers, 

when projected to the universe of callers during the week of our 

fieldwork, reported that they were satisfied overall with the customer 

service they received. (See Appendix A for confidence intervals for all 

point estimates and results of chi-square tests.) The remaining 

34 callers hung up before receiving a response to their question(s), citing 

reasons such as hold times too long or calling after business hours, and 

thus were not asked to rate satisfaction. Table 2 below shows reported 

overall satisfaction by group.

7 

Table 2: Caller Satisfaction with Customer Service 

Type of Call Center 

Percentage of Callers Reporting 

Satisfied Unsatisfied Neutral or Unsure 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

77% 

91% 

78% 

15% 

4% 

19% 

8% 

5% 

3% 

Source: Office of Inspector General analysis of caller surveys, 2004. 

We asked callers about a variety of experiences, such as: use of the IVR; 

the need to call multiple times; receipt of an answer and all the 

information they needed; actions taken as a result of the call; receipt of 

an answer quickly or being placed on hold; the courtesy of the customer 

service representative; and overall satisfaction with the customer 

service received. We found that overall satisfaction was associated with 

three experiences:

8 

7 Caller satisfaction varied by call center group. A chi-square test provides a p-value of 

0.008, significant at the 95 percent confidence level. However, using the Bonferroni 

method for testing each of the three pair-wise comparisons, a p-value of 0.0167 or less is 

necessary to have a significant difference at the 95 percent confidence level. The 

comparison between small and large call centers was significant (p-value of 0.008). 

However, the comparison between 1-800-MEDICARE and large call centers was not 

significant (p-value of 0.0169). 

8 Correlations are statistically significant at the 95 percent confidence level. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 8 
Report Template Version = 01-13-05 

F I NDING S 

o finding the IVR easy to use, 

o receiving an answer to their question and all of the information 

they needed, and 

o receiving an answer to their question as quickly as desired. 

However, 44 percent of all callers 

reported difficulty accessing 

information from call centers 

Forty-four percent of all callers 

during our week of fieldwork had 

difficulty accessing information, 

reporting at least one of the 

following experiences: (1) finding the IVR not easy to use, (2) not 

receiving an answer to their question or all the information they 

needed, or (3) not receiving an answer to their question as quickly as 

desired. A profile of callers and their experiences can be found in 

Appendix B. 

Thirty-two percent of callers encountering an IVR system reported that it 

was not easy to use 

At the time of our fieldwork, small and large call centers had the option 

of providing an IVR. According to the CMS contractor manual, “IVRs 

are intended to assist beneficiaries in obtaining information on general 

Medicare program questions, publication requests, and appeal rights.”9 

Of the small and large call centers, 42 of the 63 used an IVR, in addition 

to 1-800-MEDICARE. We asked callers who interacted with an IVR 

during our week of fieldwork if the recording that answered their call 

was easy to use. Thirty-two percent of these callers reported that the 

recording was not easy to use. Respondent experiences are shown in 

Table 3 on the next page.10 

9 The Medicare Contractor Beneficiary and Provider Communications Manual contained 

this statement concerning IVRs at the time of our review. We note that the manual was 

revised effective June 3, 2005. 

10 Differences between groups are not statistically significant at the 95 percent confidence 

level (p-value = 0.111). 

OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 9 
Report Template Version = 01-13-05 

F I NDING S 

Table 3: Caller Experience with IVR 

Type of Call Center 

Percentage of Callers Reporting 

IVR easy to use IVR not easy to use 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

71% 

74% 

60% 

29% 

26% 

40% 

Source: Office of Inspector General analysis of caller surveys, 2004. 

In addition to asking whether the IVR was easy to use, we captured 

callers’ comments about the IVR. Forty-two percent of the sampled 

callers interacting with an IVR expressed a negative opinion of the IVR. 

These callers’ comments included complaints that the message was too 

long or contained too many choices, or that the menus were not 

applicable to their questions. Callers also expressed feelings of 

frustration, irritation, confusion, or lack of understanding. One caller to 

a small call center stated that she was “. . . very unhappy with the IVR. 

It makes you want to cry.” Thirteen percent of sampled callers 

expressed the desire to speak with a live person rather than a recording. 

One caller to 1-800-MEDICARE stated, “People are always waiting to 

see what to punch if you want to talk to a person. Why don’t they make 

‘talk to a live person’ the first option?” 

As shown in Table 3, 40 percent of callers to 1-800-MEDICARE found 

the IVR not easy to use. In July 2004, 1-800-MEDICARE became the 

sole point of entry for all callers, meaning that the 1-800-MEDICARE 

IVR initially answers all calls. 

To gain firsthand experience with the 1-800-MEDICARE IVR, we called 

it on five separate occasions in August 2004. We found that, to arrive at 

an option to speak with a customer service representative, we had to 

respond to a variety of questions about the nature of the inquiry before 

we were given the option to select “agent” in order to speak with a 

representative. Answering these questions took approximately 

2 minutes. Subsequent to our fieldwork, we called the 

1-800-MEDICARE IVR and found that the option to speak with a 

representative had been placed in the main menu and took 

approximately 50 seconds to reach. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 10 
Report Template Version = 01-13-05 

F I NDING S 

Twenty-four percent of callers reported not receiving an answer to their 

question or all the information they needed 

We asked callers who received responses whether they received an 

answer to their question and whether they received all the information 

they needed. During our week of fieldwork, 24 percent of callers 

provided a negative response to one or both of these questions. 

We asked these callers to explain their answer to the questions above. 

Some callers expressed that the customer service representative could 

not answer their question or gave an explanation that left them 

confused. Others reported that they received no information or received 

incomplete information. One caller said, “I don’t know any more now 

than before I called.” Another caller remarked, “I am still confused. I 

get a different answer each time I call.” Respondent experiences are 

shown in Table 4 below.11 

Table 4: Caller Experience with Receiving Answers and 

Complete Information 

Type of Call Center 

Percentage of Callers Reporting 

Received BOTH an 

answer AND all 

information needed 

Did not receive an 

answer AND/OR all 

information needed 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

69% 

81% 

70% 

31% 

19% 

30% 

Source: Office of Inspector General analysis of caller surveys, 2004. 

Twelve percent of callers reported not receiving the answer to their question 

as quickly as they desired 

Our review of CMS’s performance metrics requirements indicated an 

emphasis on addressing caller questions and issues quickly. We asked 

callers whether they received an answer to their question as quickly as 

they desired. During our week of fieldwork, 12 percent of callers 

reported they did not. Respondent experiences are shown in Table 5 on 

the next page.12 

11 Differences between groups are not statistically significant at the 95 percent confidence 

level (p-value = 0.11). 

12 Differences between groups are statistically significant at the 95 percent confidence level 

(p-value = 0.018). All rows do not equal 100 percent due to rounding. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 11 
Report Template Version = 01-13-05 

F I NDING S 

Table 5: Caller Experience with Receiving an Answer to Their 

Question as Quickly as Desired 

Type of Call Center 

Percentage of Callers Reporting Receiving an 

Answer to their Question 

As quickly as 

desired 

Not as quickly 

as desired Unsure 

Small Call Centers 

Large Call Centers 

1-800-MEDICARE 

77% 

88% 

80% 

21% 

7% 

16% 

2% 

4% 

4% 

Source: Office of Inspector General analysis of caller surveys, 2004. 

We also captured comments from callers regarding multiple transfers 

and multiple calls to different telephone numbers. Nine percent of all 

sampled callers shared experiences such as referrals to other sources 

(e.g., telephone numbers) that did not resolve their issue or could not 

answer their question. As one caller stated, “Who to call is a secret. It 

was a round robin to get to the right telephone number.” During the 

week of our fieldwork, sampled callers made an average of 2.43 calls to 

call centers, ranging from 1 to 8. 

Callers and call center managers both 

placed a high priority on accuracy of 

answers, yet evidence suggests 

oversight of accuracy may be inadequate 

In addition to asking callers about 

their experiences during their call 

to a Medicare-funded call center, 

we also asked them about their 

priorities for customer service. 

We identified three priority areas for telephone customer service based 

upon our review of the QCM process and discussions with CMS officials: 

(1) accuracy, (2) courtesy, and (3) amount of time caller spends on the 

telephone. While we recognize that all of these elements are important 

to the customer service experience, we asked both callers and call center 

managers to rank the three elements in order of importance. 

Overall, 67 percent of all callers during our week of fieldwork ranked 

accuracy as the highest priority. Forty-nine percent of all callers 

ranked their priorities as accuracy, courtesy, and time spent on the 

telephone, while 18 percent of all callers ranked their priorities as 

accuracy, time spent on the telephone, and courtesy. We believe it is 

OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 12 
Report Template Version = 01-13-05 

F I NDING S 

important to note that 24 percent of callers reported not receiving an 

answer to their question or all the information they needed. Both of 

these experiences are related to accuracy. 

Of the call center managers who ranked their customer service 

priorities,

13 71 percent (37/52) ranked their priorities as accuracy, 

courtesy, and time spent on the telephone. The only other combination 

of priorities for call center managers, as reported by 29 percent (15/52), 

was courtesy, accuracy, and time spent on the telephone. 

Oversight of accuracy may be inadequate 

We accompanied a CMS review team on a performance evaluation in 

June 2004 and observed that the evaluation focused on validation of 

reported performance results (e.g., number of monitored calls, number 

of calls answered in less than 60 seconds, average speed of answer) and 

documented training activities. However, the review team spent little 

time assessing the accuracy of answers given to callers. 

The review protocol requires reviewers to observe and listen to eight 

calls that the call center is monitoring as part of its QCM process. In 

practice, the review team observed, but did not evaluate the accuracy of 

four calls being scored. We inquired of the review team why they only 

observed the calls without evaluating the accuracy of the answers. A 

CMS review team member told us that evaluating the accuracy of the 

calls was not feasible since the review teams do not always include staff 

with the necessary expertise. 

Our fieldwork supported GAO’s findings from a July 2004 GAO report14 

which found that the CMS performance evaluation criteria “are not 

designed to verify that [customer service representatives’] responses to 

providers are accurate.” In addition, a December 2004 GAO report15 

found that “CMS and its contractors do not emphasize [customer service 

representatives’] ability to answer questions accurately using [CMSapproved guidance].” 

 OE I-07-04-00 030 

13 One call center manager refused to rank priorities, and two call center managers 

indicated that both accuracy and courtesy were the top priority, followed by the amount 

of time spent on the telephone. 

14 GAO-02-249 “Communications with Physicians Can Be Improved,” which can be accessed 

at http://www.gao.gov/new.items/d02249.pdf. 

15 GAO-04-669 “Call Centers Need to Improve Responses to Policy-Oriented Questions from 

Providers,” which can be accessed at http://www.gao.gov/new.items/d04669.pdf. 

MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 13 
Report Template Version = 01-13-05 

F I NDING S 

Some call centers conducted quality assurance activities that exceeded 

CMS requirements, including activities that focused on accuracy 

Thirteen call center managers indicated that they had enhanced their 

QCM process by: (1) evaluating more than three calls per month per 

customer service representative; (2) tying QCM results to performance 

evaluations and raises; (3) having representatives evaluate recorded 

calls; and/or (4) having representatives participate in calibrations, 

which are meetings held across call centers to ensure all quality 

auditors score calls consistently. 

Thirteen other call center managers reported administering periodic 

knowledge tests to customer service representatives. Some call centers 

administered graded written tests completed by the representative. 

Other call centers had supervisors pose as callers and ask a question. 

The supervisor then assessed the accuracy and courtesy of the 

representative’s response. 

Four call center managers reported that they conduct both enhanced 

QCM and knowledge testing. These four call centers were all large call 

centers. The sampled callers to these call centers experienced higher 

percentages of receiving an answer to their question and all the 

information they needed when compared to all other call centers. 

Sampled callers to these call centers also reported higher satisfaction 

with the customer service they received compared to all other call 

centers. However, the callers to these four call centers comprised a 

small subset of sampled calls; the results are not statistically 

significant, yet suggest the potential that these efforts may improve 

accuracy and satisfaction. 

When we asked call center managers about QCM in general, call center 

managers reported some positive aspects of the process, including: clear 

expectations for both quality auditors and customer service 

representatives, random selection of calls for monitoring, and 

identification of training needs for the representatives and the entire 

call center based on errors found in monitoring. In fact, 27 percent of 

call center managers (15/55) supported more frequent call monitoring, 

believing the minimum number of calls per month required by CMS for 

each representative did not give an adequate assessment of a 

representative’s performance. 

OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 14 
Report Template Version = 01-13-05 

ΔF I NDING S  RECOMME NDA TIONS 

Medicare telephone customer service is an important source of 

information regarding Medicare benefits for beneficiaries and their 

representatives. Their informational needs will undoubtedly increase 

as the significant program changes contained in the MMA are 

implemented. Furthermore, section 921 of the MMA specifically calls 

for prompt responses to inquiries from Medicare contractors, and for 

contractors to monitor the accuracy, consistency, and timeliness of the 

information they provide. Although our inspection found that 

84 percent of callers asked to rate satisfaction reported they were 

satisfied overall with the customer service received, 44 percent of callers 

reported difficulty accessing information from call centers. The 

response completeness and IVRs contributed to the reported difficulties. 

We recommend that CMS: 

o Strengthen current oversight to place greater emphasis on 

completeness of responses, greater efficiency for the caller, and 

accuracy of answers given by customer service representatives. 

CMS could achieve this by: including CMS staff with expertise in 

call handling on national review teams for performance 

evaluations, requiring call centers to conduct periodic knowledge 

testing with minimum passing scores, and/or increasing the 

number of calls monitored in the QCM process. Given that some 

call centers are currently engaged in conducting periodic 

knowledge testing and increased monitoring, CMS may want to 

determine the impact these practices may have on quality 

outcomes. 

o Continue to seek ways to improve the national IVR system. We 

note that subsequent to our fieldwork, CMS moved the option to 

speak with a customer service representative to the main menu, 

thereby decreasing the amount of time it required to arrive at this 

option from approximately 2 minutes to 50 seconds. 

AGENCY COMMENTS 

In its comments to the draft report, CMS expressed appreciation for the 

Office of Inspector General’s analysis and recommendations to help 

improve the benficiary inquiry call centers. CMS stated it will continue 

with quality assurance activities, some of which we suggested in our 

recommendations, already underway at the 1-800-MEDICARE call 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 15 
Report Template Version = 01-13-05 

R E C O MMENDA T I O N S 

center. CMS described these activities in detail. However, CMS does 

not plan to invest significant resources into reengineering quality 

assurance activities at the fee-for-service call centers because this 

workload will be integrated into the 1-800-MEDICARE operation. In 

addition, CMS stated that an expert in the field of IVR is being 

consulted to assist with improving and enhancing the system. For 

CMS’s complete comments, see page 17 of this report. 

OFFICE OF INSPECTOR GENERAL RESPONSE 

We appreciate CMS’s comments to this report and note that CMS is 

taking action to address issues raised in this report. We acknowledge 

that CMS revised its Medicare Contractor Beneficiary and Provider 

Communications Manual in June 2005. Changes were made to the 

report to reflect technical comments received from CMS. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 16 
Report Template Version = 01-13-05 

Δ AGENCY COMMENTS 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 17 
Report Template Version = 01-13-05 

A G E N C Y CO MMENT S

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 18 
Report Template Version = 01-13-05 

A G E N C Y CO MMENT S

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 19 
Report Template Version = 01-13-05 

A G E N C Y CO MMENT S

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 20 
Report Template Version = 01-13-05 

A G E N C Y CO MMENT S

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 21 
Report Template Version = 01-13-05 

A G E N C Y CO MMENT S

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 22 
Report Template Version = 01-13-05 

A G E N C Y CO MMENT S

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 23 
Report Template Version = 01-13-05 

ΔAPP E N D I X ~A  APPENDIX ~ A 

Confidence Intervals for Key Estimates and Chi-Square Tests 

Caller Experiences 

Finding 

Degrees of 

Freedom P-value 

Weighted Chi-Square Test 

Relationship between finding the interactive voice response easy to use and 

satisfaction 

Interactive voice response easy to 

use/Don't remember 

Interactive voice response not 

easy to use 

Percent Satisfied 

89.14 

72.99 

Percent Unsatisfied* 

10.86 

27.01 

1 0.012 

Relationship between receiving an answer to a question and all information 

needed, and satisfaction 

Received answer 

Did not receive answer 

Percent Satisfied 

95 

51.91 

Percent Unsatisfied* 

5 

48.09 

1 less than 

0.001 

Relationship between receiving an answer to a question as quickly as desired and 

satisfaction 

Received answer quickly 

Did not receive answer quickly 

Percent Satisfied 

92.06 

30.77 

Percent Unsatisfied* 

7.94 

69.23 

1 less than 

0.001 

*Includes sampled callers who reported that they were unsure or neither satisfied nor unsatisfied, in addition to 

sampled callers who reported that they were unsatisfied. 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 24 
Report Template Version = 01-13-05 

APP E N D I X ~A 

Caller Experiences 

Finding Point Estimate 

Confidence 

Interval 

95 Percent 

Callers to 1-800-MEDICARE who reported that the interactive 

voice response was not easy to use 

Callers encountering an interactive voice response who 

reported that it was not easy to use 

Callers who reported not receiving an answer to their question 

or all the information that they needed 

Callers who reported not receiving an answer as quickly as they 

desired 

Callers asked to rate satisfaction who were satisfied overall with 

the customer service received 

Callers who reported difficulty accessing information from call 

centers 

84.5% 

43.7% 

31.6% 

39.7% 

24.4% 

12.4% 

80.1% - 88.9% 

37.7% - 49.6% 

25.9% - 37.4% 

30.1% - 49.4% 

18.9% - 29.9% 

8.3% - 16.4% 

Customer Service Priorities 

Finding Point Estimate 

Confidence 

Interval 

95 Percent 

Callers who ranked priorities as: 1) accuracy, 2) time spent on 

the telephone, and 3) courtesy 

Callers who ranked priorities as: 1) accuracy, 2) courtesy, and 

3) time spent on the telephone 48.7% 

18.1% 

42.5% - 54.9% 

13.4% - 22.8% 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 25 
Report Template Version = 01-13-05 

Δ APPENDIX ~ B 

Profile of Surveyed Callers 

Small Call 

Centers 

Large Call 

Centers 

1-800-

MEDICARE Total 

Who called? 

 Beneficiary* 

 Beneficiary’s family member 

 Other (e.g., provider or insurance office staff) 

69 

27 

4 

70 

32 

3 

81 

15 

4 

220/305 

74/305 

11/305 

Why did they call? 

 Medical bill/summary notice 

Benefit question 

 Other (e.g., address change, multiple issues) 

56 

19 

25 

56 

25 

24 

19 

48 

33 

131/305 

92/305 

82/305 

How did they know what phone number to call? 

 Printed on a notice I received 

 Someone gave me the number 

 Medicare & You handbook 

 Previous knowledge 

 1-800-MEDICARE advertisement 

 Other (e.g., Medicare card, telephone book) 

51 

14 

5 

3 

3 

24 

61 

21 

5 

8 

0 

10 

36 

8 

19 

2 

6 

29 

148/305 

43/305 

29/305 

13/305 

9/305 

63/305 

Did they try to get their question answered through 

any other source before calling? 

 No other source consulted 

 Medicare handbook or brochure 

 Internet Web site** 

 Friend, family member, counselor 

 Other (e.g., call to provider, secondary insurance) 

72 

3 

1 

1 

23 

78 

0 

1 

0 

26 

70 

2 

1 

1 

26 

220/305 

5/305 

3/305 

2/305 

75/305 

How many times did they call a Medicare-funded call 

center during the week? 

1-3 

4-8 

 Average number of calls 

79 

21 

2.52 

88 

17 

2.34 

83 

17 

2.44 

250/305 

55/305 

2.43 

How did callers describe CSR***/recording pace, 

volume, and clarity?

 CSR/recording spoke at the right pace 

 CSR/recording spoke at the right volume 

 CSR/recording spoke clearly 

85 

84 

88 

93 

93 

95 

77 

82 

80 

255/271 

259/271 

263/271 

How did callers rate their experience with the 

customer service received? 

Satisfied 

Unsatisfied 

 Neither or Unsure 

70 

14 

8 

88 

4 

4 

64 

16 

3 

222/271 

34/271 

15/271 

*These totals include four callers who called on behalf of themselves and for family members. Two of these callers were in the

large call center groups, and two callers were in the 1-800-MEDICARE group. 

**One caller in the 1-800-MEDICARE group asked a friend, family member, or counselor in addition to accessing an Internet 

Web site. 

***Customer service representative (CSR).

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 26 
Report Template Version = 01-13-05 

Δ ACKNOWLEDGMENTS 

This report was prepared under the direction of Brian T. Pattison, 

Regional Inspector General for Evaluation and Inspections in the 

Kansas City regional office, and Gina C. Maree, Assistant Regional 

Inspector General. Other principal Office of Evaluation and Inspections 

staff who contributed include: 

Tricia Fields, Team Leader 

Steven Milas, Program Analyst 

Linda Paddock, Program Analyst 

Elander Phillips, Program Analyst 

Perry Seaton, Program Analyst 

Deborah Walden, Program Analyst 

Tricia Davis, Director, Medicare & Medicaid Branch 

Technical Assistance 

Robert Gibbons, Program Analyst 

 OE I-07-04-00 030 MEDI C ARE BENEF I C IARY T E L E P H O N E CUST O M E R S ERV I CE 27