Report Template Version  = 01-13-05  
Department of Health and Human Services

OFFICE OF 

INSPECTOR GENERAL 

M B
EDICARE  ENEFICIARY 

T C
ELEPHONE  USTOMER 

S
ERVICE

Daniel R. Levinson

Inspector General 

August 2005

OEI-07-04-00030
 Report Template Version  = 01-13-05  
Office of Inspector General 

http://oig.hhs.gov 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, 
as amended, is to protect the integrity of the Department of Health and Human Services 
(HHS) programs, as well as the health and welfare of beneficiaries served by those 
programs.  This statutory mission is carried out through a nationwide network of audits, 
investigations, and inspections conducted by the following operating components: 
Office of Audit Services 
The OIG's Office of Audit Services (OAS) provides all auditing services for HHS, either by 
conducting audits with its own audit resources or by overseeing audit work done by others. 
Audits examine the performance of HHS programs and/or its grantees and contractors 
in carrying out their respective responsibilities and are intended to provide independent 
assessments of HHS programs and operations in order to reduce waste, abuse, and 
mismanagement and to promote economy and efficiency throughout the department. 
Office of Evaluation and Inspections 
The OIG's Office of Evaluation and Inspections (OEI) conducts management and 
program evaluations (called inspections) that focus on issues of concern to the 
department, the Congress, and the public. The findings and recommendations contained 
in the inspections reports generate rapid, accurate, and up-to-date information on the 
efficiency, vulnerability, and effectiveness of departmental programs.  The OEI also 
oversees State Medicaid fraud control units, which investigate and prosecute fraud and 
patient abuse in the Medicaid program. 
Office of Investigations 
The OIG's Office of Investigations (OI) conducts criminal, civil, and administrative 
investigations of allegations of wrongdoing in HHS programs or to HHS beneficiaries 
and of unjust enrichment by providers.  The investigative efforts of OI lead to criminal 
convictions, administrative sanctions, or civil monetary penalties. 
Office of Counsel to the Inspector General 
The Office of Counsel to the Inspector General (OCIG) provides general legal services to 
OIG, rendering advice and opinions on HHS programs and operations and providing all 
legal support in OIG's internal operations.  The OCIG imposes program exclusions and 
civil monetary penalties on health care providers and litigates those actions within the 
department.  The OCIG also represents OIG in the global settlement of cases arising 
under the Civil False Claims Act, develops and monitors corporate integrity agreements, 
develops compliance program guidances, renders advisory opinions on OIG sanctions to 
the health care community, and issues fraud alerts and other industry guidance.  Report Template Version  = 01-13-05 
  E X E C U T I V E   S U M M A R Y  

OBJECTIVE 
To assess callers experiences with Medicare-funded call centers and to 
determine priorities for and efforts to ensure quality customer service. 
BACKGROUND 
The Centers for Medicare & Medicaid Services (CMS) serves the 
informational needs of approximately 40 million Medicare beneficiaries 
and/or their representatives by using multiple communication sources. 
The most heavily used communication sources are the Medicare-funded 
call centers. At the time of our fieldwork, beneficiaries could call either 
a fee-for-service contractor call center or 1-800-MEDICARE with 
questions about Medicare.  There were 29.3 million calls to 
Medicare-funded call centers in fiscal year 2004. 
We administered a survey to 305 callers over a 1-week period to assess 
their experiences with Medicare telephone customer service.  We asked 
callers if they were satisfied with the customer service they received, if 
they believed their questions were answered, and if they received all the 
information they needed.  We also asked callers about their priorities 
for customer service.  We administered a survey to the managers of all 
the call centers to determine the call center managers priorities for 
quality customer service and to identify any quality improvement efforts 
planned or underway.  Finally, we reviewed CMSs oversight activities 
related to call centers, including accompanying CMS staff on a 
performance evaluation of a large fee-for-service call center. 
FINDINGS 
Eighty-four percent of callers asked to rate satisfaction were 
satisfied overall with the customer service they received; however, 
44 percent of all callers reported difficulty accessing information 
from call centers.  The 44 percent of callers who had difficulty 
accessing information reported at least one of the following experiences: 
(1) finding the Interactive Voice Response not easy to use, (2) not 
receiving an answer to their question or all the information they 
needed, or (3) not receiving the answer to their question as quickly as 
desired. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  i  Report Template Version  = 01-13-05 
EE XX EE CC UU TT II VV EE    SS UU MM MM AA RR YY
Callers and call center managers both placed a high priority on 
accuracy of answers, yet evidence suggests oversight of accuracy 
may be inadequate. Sixty-seven percent of all callers and 71 percent of 
call center managers ranked accuracy as their highest priority. 
Although we did not test accuracy of answers specifically, we believe it 
is important to note that 24 percent of callers reported not receiving an 
answer to their question or all the information they needed. Both of 
these experiences are related to accuracy. Findings in two Government 
Accountability Office (GAO) reviews of Medicare-funded call centers 
raised concerns that oversight of accuracy may be inadequate. A 
July 2004 GAO report found that the performance evaluation criteria 
. . . are not designed to verify that [customer service representatives] 
responses to providers are accurate. A December 2004 GAO report 
found that CMS and its contractors do not emphasize [customer service 
representatives] ability to answer questions accurately using [CMS-
approved guidance]. While limited, our fieldwork supported GAOs 
findings. The CMS performance evaluation we observed focused on 
validation of reported performance results (e.g., number of monitored 
calls, number of calls answered in less than 60 seconds, average speed of 
answer) and documented training activities. However, the review team 
spent little time assessing the accuracy of answers given to callers. 
Some call centers conducted quality assurance activities that focused on 
accuracy, exceeding CMS requirements. These activities included 
increasing the number of monitored calls in the Quality Call Monitoring 
process, tying monitored performance to customer service 
representative evaluations, involving representatives in the 
administration of quality assurance activities, and administering 
periodic knowledge tests. 
RECOMMENDATIONS 
We recommend that CMS: 
o 	 Strengthen current oversight to place greater emphasis on 
completeness of responses, greater efficiency for the caller, and 
accuracy of answers given by customer service representatives. 
CMS could achieve this by: (1) including CMS staff with expertise 
in call handling on national review teams for performance 
evaluations, (2) requiring call centers to conduct periodic 
knowledge testing with minimum passing scores, and/or 
(3) increasing the number of calls monitored in the Quality Call 
OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  ii  Report Template Version  = 01-13-05 
E X E C U T I V E   S U M M A R Y  
Monitoring process.  Given that some call centers are currently 
engaged in conducting periodic knowledge testing and increased 
monitoring, CMS may want to determine the impact these 
practices may have on quality outcomes. 
o 	 Continue to seek ways to improve the national Interactive Voice 
Response system.  We note that subsequent to our fieldwork, CMS 
moved the option to speak with a customer service representative 
to the main menu, thereby decreasing the amount of time it 
required to arrive at this option from approximately 2 minutes to 
50 seconds.   
AGENCY COMMENTS 
In its comments to the draft report, CMS expressed appreciation for 
the Office of Inspector Generals analysis and recommendations to 
help improve the benficiary inquiry call centers. CMS stated it will 
continue with quality assurance activities, some of which we 
suggested in our recommendations, already underway at the 
1-800-MEDICARE call center.  CMS described these activities in 
detail.  However, CMS does not plan to invest significant resources 
into reengineering quality assurance activities at the fee-for-service 
call centers because this workload will be integrated into the 
1-800-MEDICARE operation.  In addition, CMS stated that an expert 
in the field of interactive voice response is being consulted to assist 
with improving and enhancing the system.  For CMSs complete 
comments, see page 17 of this report. 
OFFICE OF INSPECTOR GENERAL RESPONSE 
We appreciate CMSs comments to this report and note that CMS is 
taking action to address issues raised in this report.  Changes were 
made to the report to reflect technical comments received from CMS. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  iii  Report Template Version  = 01-13-05  
  T A B L E   O F   C O N T E N T S  

EXECUTIVE SUMMARY ........................................  i

INTRODUCTION ................................................ 1 

FINDINGS ....................................................... 8 

Positiveoverall caller satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . .  8

Difficulty accessing information. . . . . . . . . . . . . . . . . . . . . . . . . . . .  9

Priorities and quality assurance activities. . . . . . . . . . . . . . . . . . .  12

RECOMMENDATIONS ......................................... 15 

AGENCY COMMENTS ......................................... 17

APPENDIXES .................................................. 24 

A: 	Confidence Intervals for Key Estimates and Chi-Square

  Tests  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  24

B:  Profileof Surveyed Callers . . . . . . . . . . . . . . . . . . . . . . . . . . . .  26

ACKNOWLEDGMENTS ......................................... 27
 Report Template Version  = 01-13-05 
  I N T R O D U C T I O N  

OBJECTIVE 
To assess callers experiences with Medicare-funded call centers and to 
determine priorities for and efforts to ensure quality customer service. 
BACKGROUND 
Beneficiary Telephone Inquiries 
The Centers for Medicare & Medicaid Services (CMS) serves the 
informational needs of approximately 40 million Medicare beneficiaries 
and/or their representatives by using multiple communication sources. 
The most heavily used communication sources are the Medicare-funded 
call centers. At the time of our fieldwork, beneficiaries could call either 
a fee-for-service contractor call center or 1-800-MEDICARE (which is 
also a CMS contractor) with questions about Medicare.  Fee-for-service 
contractors (carriers, fiscal intermediaries, and durable medical 
equipment regional carriers) operated 63 separate call centers.1 In 
fiscal year (FY) 2004, the fee-for-service call centers handled 
approximately 12.8 million calls at a cost of $96.3 million, while 
1-800-MEDICARE handled 16.5 million calls at a cost of $104.2 million. 
Since Medicares inception in 1965, Medicare contractors have operated 
the telephone customer service system as a group of stand-alone call 
centers.  The Balanced Budget Act of 1997 required a toll-free number 
for inquiries regarding the newly created Medicare+Choice program. In 
1998, CMS established 1-800-MEDICARE for this purpose.  In 1999, the 
phone number was phased in as part of the National Medicare 
Education Program and offered general information about Medicare, 
health plan options, supplemental insurance, and referral telephone 
numbers for help with claims or more complex issues.  The original 
purpose of 1-800-MEDICARE, a source of information about 
Medicare+Choice options, broadened as new initiatives began.  
Section 923(d) of the Medicare Prescription Drug, Improvement, and 
Modernization Act of 2003 (MMA) officially established 1-800-
MEDICARE as the primary source of general Medicare information and 
assistance. In July 2004, 1-800-MEDICARE became the single point of 
entry for the telephone customer service system, with transfer 
capabilities to the various fee-for-service call centers. 
1 CMS provided a listing of 63 fee-for-service call centers dated April 2, 2004. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  1  Report Template Version  = 01-13-05 
I N T R O D U C T I O N  
CMS Goals for Customer Service Satisfaction 
According to the CMS Web site, CMSs customer service mission is to 
continuously improve Medicare beneficiary customer satisfaction 
through the delivery of high quality and cost effective customer service. 
CMSs FY 2004 Government Performance and Results Act Annual 
Performance Plan indicated that the performance goal for telephone 
customer service was to improve beneficiary telephone customer service 
in terms of accessibility, accuracy of response, and caller satisfaction.  
During FYs 2000-2002, CMS developed baselines and collected data 
related to this performance goal.  However, in the plan narrative 
discussing the performance results, CMS indicated that in 
FYs 2003-2004, it redirected efforts and resources away from the 
performance goal of improving beneficiary telephone customer service in 
terms of accessibility, accuracy of response, and caller satisfaction 
(including funding for a national caller satisfaction survey), instead 
focusing on creating a single point of entry (1-800-MEDICARE).  Thus, 
no current national baseline for beneficiary satisfaction exists. 
In FY 2004, only 3 of the 63 fee-for-service call centers conducted 
beneficiary satisfaction surveys, which were part of a performance 
incentive pilot program.  At the time of our fieldwork, CMS required its 
1-800-MEDICARE contractor to conduct 400 beneficiary satisfaction 
surveys per month.   
CMS Requirements for Call Centers 
The Medicare Contractor Beneficiary and Provider Communications 
Manual outlines the guidelines for telephone customer service, 
including: hours of operation, bilingual services, services for people 
with hearing impairments, and requirements related to Interactive 
Voice Response (IVR).2,3  Certain performance metrics must be met in 
terms of how quickly calls are answered, how quickly callers who choose 
to speak with a customer service representative actually speak with a 
representative, how often callers questions are answered on the initial 
call, and how often callers receive a busy signal. 
2 An IVR is a software application in which someone uses a touch-tone telephone to interact 
with a database to acquire information.  An IVR provides prerecorded voice responses 
based on voice or keypad input. 
3 Medicare Contractor Beneficiary and Provider Communications Manual, Chapter 2, 
section 20.1. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  2  Report Template Version  = 01-13-05 
I N T R O D U C T I O N  
CMS requires all contractors to use Quality Call Monitoring (QCM) to 
monitor, measure, and report the quality of service they provide.4  The 
QCM is the primary method CMS uses to assess whether call centers 
are meeting established performance standards.   
The QCM process requires a quality auditor to monitor a minimum of 
three calls per customer service representative per month at 
fee-for-service call centers, and four calls per representative per month 
at 1-800-MEDICARE.  The quality auditor uses a scorecard to rate the 
representative in areas such as adherence to the Privacy Act, customer 
skills (e.g., greeting, tone, volume), and knowledge skills (e.g., accuracy, 
completeness, call action).  The following performance standards must 
be met for the calls monitored each month through the QCM: 
o 	 The percentage of customer service representatives scoring as 
Pass for adherence to the Privacy Act should be no less than 
90 percent. 
o 	 The percentage of customer service representatives scoring as 
Achieves Expectation or higher for customer skills should be no 
less than 90 percent. 
o 	 The percentage of customer service representatives scoring as 
Achieves Expectation or higher for knowledge skills should be no 
less than 90 percent. 
The results of the scorecards are used for coaching the representatives. 
Scorecard results are aggregated and reported to CMS on a monthly 
basis. 
CMS Evaluation of Call Centers 
CMS validates the reported scorecard data during a fee-for-service 
contractors performance evaluation.  According to the FY 2004 review 
protocol, the purpose of the performance evaluation is to determine 
whether the contractor is:  (1) answering calls from Medicare 
beneficiaries in an efficient, accurate, and professional manner that 
meets established performance standards; (2) reporting performance 
data accurately to CMS; and (3) following established guidelines for 
training and coaching customer service representatives to properly 
respond to telephone inquiries. Based on FYs 2001-2003 contractor 
performance evaluation reports and the schedule of FY 2004 planned 
4 Medicare Contractor Beneficiary and Provider Communications Manual, Chapter 2, 
section 20.1.7. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  3  Report Template Version  = 01-13-05 
I N T R O D U C T I O N  
evaluations that CMS provided to us, we determined that CMS 
conducted 31 performance evaluations in FYs 2001-2004 at 
23 fee-for-service call centers. 
CMS employs different evaluation methods for 1-800-MEDICARE.  
Every 4 months, the 1-800-MEDICARE contractor provides a narrative 
self-evaluation that CMS uses to conduct an assessment of: 
(1) performance and progress achieved in various functional areas 
(e.g., call center operations, information technology, response to 
requests for publications); (2) a comparison of actual performance to the 
standard for various performance metrics; and (3) a rating for contract 
compliance.  In addition, a quality assurance contractor monitors the 
operation of the IVR and calls answered by customer service 
representatives on a monthly basis. 
Other Studies 
In a February 2002 study for which CMS contracted with a private 
consultant, beneficiaries who had contacted any one of the eight largest 
Medicare contractors with a billing question that was transferred to 
their respective fraud units were surveyed.  Seventy-five percent of 
beneficiaries reported being satisfied with their overall experience.  In a 
February 2002 report regarding provider inquiries entitled 
Communications with Physicians can be Improved, (GAO-02-249), the 
Government Accountability Office (GAO) found that the customer 
service representatives they tested provided complete and accurate 
answers to their questions 15 percent of the time. In a July 2004 
followup report entitled Call Centers Need to Improve Responses to 
Policy-Oriented Questions from Providers, (GAO-04-669), GAO found 
that only 4 percent of answers given were complete and accurate. 
Section 923 of the MMA mandated GAO to conduct a study of the 
accuracy and consistency of information provided through 
1-800-MEDICARE.  This study, Accuracy of Responses from the 
1-800-MEDICARE Help Line Should Be Improved, was released in 
December 2004 (GAO-05-130).  GAO reviewers called 
1-800-MEDICARE and posed test questions to the representatives 
regarding the Medicare program. GAO found that the representatives 
provided accurate answers 61 percent of the time. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  4  Report Template Version  = 01-13-05 
I N T R O D U C T I O N

METHODOLOGY 
We used three methods to gather information for this inspection: 
(1) telephone surveys with callers to fee-for-service or 
1-800-MEDICARE call centers, (2) a review of CMS oversight activities, 
and (3) interviews with call center managers. 
Telephone Surveys 
We divided the call centers into three groups:  large call centers, small 
call centers, and 1-800-MEDICARE.  Large and small designations were 
determined by calculating the mean FY 2003 call volume per 
fee-for-service call center (223,609 calls).  Fee-for-service call centers 
with FY 2003 call volumes below the mean were designated as small, 
and call centers with FY 2003 call volumes above the mean were 
designated as large.  The FY 2003 call volume range for small call 
centers was 4,531 to 223,396 calls and the FY 2003 call volume range 
for large call centers was 241,066 to 1,118,851 calls.  We obtained a 
telephone network generated listing of all calls made to small and large 
call centers and 1-800-MEDICARE call centers during the week of 
April 12-16, 2004.5  The listing included the following information:  call 
center telephone number dialed, originating telephone number, and 
date and time of call. We spoke with CMS officials to ensure that the 
selected week was typical of weeks throughout the year. 
For each of 5 days (Monday through Friday), we eliminated duplicate 
originating telephone numbers (some callers called more than once in a 
given day) and randomly selected 75 callers.  For each day, these callers 
were divided into 3 groups:  25 who called small call centers, 25 who 
called large call centers, and 25 who called 1-800-MEDICARE 
(75 x 5 days = 375 randomly selected callers) for a total of 15 strata.  As 
illustrated in Table 1 on the next page, we completed 305 telephone 
surveys from the sample of 375 callers, for an 81 percent response rate. 
Ninety-seven percent of our surveys were completed within 2 business 
days of the callers sampled call to the call centers.  The other 3 percent 
were completed within 5 business days.  We administered our survey to 
determine callers experiences, such as whether callers believed their 
questions were answered, whether callers received all the information 
they needed, and how they rated their experience based on selected 
5 Small and large call centers operate Monday through Friday during typical business 
hours; 1-800-MEDICARE operates 7 days a week, 24 hours per day. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  5  Report Template Version  = 01-13-05 
I N T R O D U C T I O N  
QCM performance standards.  We did not ask 34 callers all survey 
questions because they ended their calls when they were unable to 
receive a response to their question from a customer service 
representative or IVR (e.g., called after business hours); therefore, 
271 callers were asked to rate their experiences related to the responses 
they received from the call center. 
Table 1:  Population, Sampled Calls, and Completed Surveys by Strata 
Day of Week  Type of Call Center  Population  Sampled Calls  Completed Surveys 
Small Call Centers  15,309  25  22 
Monday  Large Call Centers  47,450  25  19 
1-800-MEDICARE  32,716  25  20 
Small Call Centers  12,544  25  22 
Tuesday  Large Call Centers  40,277  25  22 
1-800-MEDICARE  29,262  25  21 
Small Call Centers  11,550  25  20 
Wednesday  Large Call Centers  35,963  25  22 
1-800-MEDICARE  27,811  25  19 
Small Call Centers  11,113  25  15 
Thursday  Large Call Centers  36,487  25  22 
1-800-MEDICARE  29,852  25  21 
Small Call Centers  10,769  25  21 
Friday  Large Call Centers  35,131  25  20 
1-800-MEDICARE  31,029  25  19 
Total  407,263  375  305 
Source:  Office of Inspector General analysis of telephone network data, 2004. 
All 305 callers were asked questions about their priorities for customer 
service and their experiences during their calls.  We performed 
comparisons with regard to overall customer service satisfaction, 
finding the IVR easy to use, receiving an answer to the callers question 
and all of the information needed, and receiving an answer to the 
callers question as quickly as desired.  We analyzed results with respect 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  6  Report Template Version  = 01-13-05 
II NN TT RR OO DD UU CC TT II OO NN
to the three groups: callers to small call centers, callers to large call 
centers, and callers to 1-800-MEDICARE. Findings are projected to the 
week of April 12-16, 2004. 
We did not administer test questions to customer service 
representatives to directly test the accuracy of answers given to callers. 
However, we did ask callers if they received an answer to their question 
and whether they received all of the information they needed. We chose 
not to directly test accuracy due to the December 2004 GAO report 
addressing this issue. To gain firsthand experience with the 
1-800-MEDICARE IVR, we called it on five separate occasions in 
August 2004. 
Review of CMS Oversight Activities 
We obtained the date and results of all call center performance 
evaluations conducted in FY 2001, FY 2002, and FY 2003, as well as 
improvement plans, action steps, progress, and followup of the results. 
We accompanied CMS staff on a performance evaluation of a large call 
center in June 2004. We observed the sampling methods, review 
procedures, and protocols used in conducting the performance 
evaluation. 
Structured Interviews with Call Center Managers 
We conducted a total of 55 telephone interviews with managers from all 
of the small and large call centers and 1-800-MEDICARE call centers to 
determine the call center managers priorities for quality customer 
service and to identify any quality improvement efforts planned or 
underway.6 We asked for results data and opinions regarding these 
efforts. 
Standards 
This inspection was conducted in accordance with the Quality 
Standards for Inspections issued by the Presidents Council on 
Integrity and Efficiency. 
6 Some large fee-for-service contractors operated multiple call centers. We combined some 
interviews, meaning that the total number of interviews conducted is less than the total 
number of call centers. 
OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  7  Report Template Version  = 01-13-05 
  F I N D I N G S

Of the 305 callers with whom we 
Eighty-four percent of callers asked to rate 
spoke, 271 received responses to 
satisfaction were satisfied overall with the 
their questions from customer 
customer service they received  service representatives or IVRs.  
Eighty-four percent of these callers, 
when projected to the universe of callers during the week of our 
fieldwork, reported that they were satisfied overall with the customer 
service they received.  (See Appendix A for confidence intervals for all 
point estimates and results of chi-square tests.)  The remaining 
34 callers hung up before receiving a response to their question(s), citing 
reasons such as hold times too long or calling after business hours, and 
thus were not asked to rate satisfaction.  Table 2 below shows reported 
overall satisfaction by group.7 
Table 2:  Caller Satisfaction with Customer Service 
Percentage of Callers Reporting 
Type of Call Center 
Satisfied  Unsatisfied  Neutral or Unsure 
Small Call Centers  77%  15%  8% 
Large Call Centers  91%  4%  5% 
1-800-MEDICARE  78%  19%  3% 
Source:  Office of Inspector General analysis of caller surveys, 2004. 
We asked callers about a variety of experiences, such as:  use of the IVR; 
the need to call multiple times; receipt of an answer and all the 
information they needed; actions taken as a result of the call; receipt of 
an answer quickly or being placed on hold; the courtesy of the customer 
service representative; and overall satisfaction with the customer 
service received.  We found that overall satisfaction was associated with 
three experiences:8 
7 Caller satisfaction varied by call center group.  A chi-square test provides a p-value of 
0.008, significant at the 95 percent confidence level.  However, using the Bonferroni 
method for testing each of the three pair-wise comparisons, a p-value of 0.0167 or less is 
necessary to have a significant difference at the 95 percent confidence level.  The 
comparison between small and large call centers was significant (p-value of 0.008). 
However, the comparison between 1-800-MEDICARE and large call centers was not 
significant (p-value of 0.0169). 
8 Correlations are statistically significant at the 95 percent confidence level. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  8  Report Template Version  = 01-13-05 
FF	 II NN DD II NN GG SS  
o 	 finding the IVR easy to use, 
o 	 receiving an answer to their question and all of the information 
they needed, and 
o 	 receiving an answer to their question as quickly as desired. 
Forty-four percent of all callers 
However, 44 percent of all callers 
during our week of fieldwork had 
reported difficulty accessing 
difficulty accessing information, 
information from call centers 
reporting at least one of the 
following experiences: (1) finding the IVR not easy to use, (2) not 
receiving an answer to their question or all the information they 
needed, or (3) not receiving an answer to their question as quickly as 
desired. A profile of callers and their experiences can be found in 
Appendix B. 
Thirty-two percent of callers encountering an IVR system reported that it 
was not easy to use 
At the time of our fieldwork, small and large call centers had the option 
of providing an IVR. According to the CMS contractor manual, IVRs 
are intended to assist beneficiaries in obtaining information on general 
Medicare program questions, publication requests, and appeal rights.9 
Of the small and large call centers, 42 of the 63 used an IVR, in addition 
to 1-800-MEDICARE. We asked callers who interacted with an IVR 
during our week of fieldwork if the recording that answered their call 
was easy to use. Thirty-two percent of these callers reported that the 
recording was not easy to use. Respondent experiences are shown in 
Table 3 on the next page.10 
9 The Medicare Contractor Beneficiary and Provider Communications Manual contained 
this statement concerning IVRs at the time of our review. We note that the manual was 
revised effective June 3, 2005. 
10 Differences between groups are not statistically significant at the 95 percent confidence 
level (p-value = 0.111). 
OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  9  Report Template Version  = 01-13-05 
F I N D I N G S  

Table 3:  Caller Experience with IVR 
Percentage of Callers Reporting 
Type of Call Center 
IVR easy to use  IVR not easy to use 
Small Call Centers  71%  29% 
Large Call Centers  74%  26% 
1-800-MEDICARE  60%  40% 
Source:  Office of Inspector General analysis of caller surveys, 2004. 
In addition to asking whether the IVR was easy to use, we captured 
callers comments about the IVR. Forty-two percent of the sampled 
callers interacting with an IVR expressed a negative opinion of the IVR.  
These callers comments included complaints that the message was too 
long or contained too many choices, or that the menus were not 
applicable to their questions.  Callers also expressed feelings of 
frustration, irritation, confusion, or lack of understanding.  One caller to 
a small call center stated that she was . . . very unhappy with the IVR. 
It makes you want to cry.  Thirteen percent of sampled callers 
expressed the desire to speak with a live person rather than a recording. 
One caller to 1-800-MEDICARE stated, People are always waiting to 
see what to punch if you want to talk to a person.  Why dont they make 
talk to a live person the first option? 
As shown in Table 3, 40 percent of callers to 1-800-MEDICARE found 
the IVR not easy to use.  In July 2004, 1-800-MEDICARE became the 
sole point of entry for all callers, meaning that the 1-800-MEDICARE 
IVR initially answers all calls.   
To gain firsthand experience with the 1-800-MEDICARE IVR, we called 
it on five separate occasions in August 2004.  We found that, to arrive at 
an option to speak with a customer service representative, we had to 
respond to a variety of questions about the nature of the inquiry before 
we were given the option to select agent in order to speak with a 
representative.  Answering these questions took approximately 
2 minutes. Subsequent to our fieldwork, we called the 
1-800-MEDICARE IVR and found that the option to speak with a 
representative had been placed in the main menu and took 
approximately 50 seconds to reach. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  10  Report Template Version  = 01-13-05 
F I N D I N G S  
Twenty-four percent of callers reported not receiving an answer to their 
question or all the information they needed 
We asked callers who received responses whether they received an 
answer to their question and whether they received all the information 
they needed.  During our week of fieldwork, 24 percent of callers 
provided a negative response to one or both of these questions.   
We asked these callers to explain their answer to the questions above. 
Some callers expressed that the customer service representative could 
not answer their question or gave an explanation that left them 
confused.  Others reported that they received no information or received 
incomplete information.  One caller said, I dont know any more now 
than before I called.  Another caller remarked, I am still confused.  I 
get a different answer each time I call.  Respondent experiences are 
shown in Table 4 below.11 
Table 4:  Caller Experience with Receiving Answers and 
Complete Information 
Percentage of Callers Reporting 
Type of Call Center  Received BOTH an  Did not receive an 
answer AND all  answer AND/OR all 
information needed  information needed 
Small Call Centers  69%  31% 
Large Call Centers  81%  19% 
1-800-MEDICARE  70%  30% 
Source:  Office of Inspector General analysis of caller surveys, 2004. 
Twelve percent of callers reported not receiving the answer to their question 
as quickly as they desired 
Our review of CMSs performance metrics requirements indicated an 
emphasis on addressing caller questions and issues quickly. We asked 
callers whether they received an answer to their question as quickly as 
they desired.  During our week of fieldwork, 12 percent of callers 
reported they did not.  Respondent experiences are shown in Table 5 on 
the next page.12 
11 Differences between groups are not statistically significant at the 95 percent confidence 
level (p-value = 0.11). 
12 Differences between groups are statistically significant at the 95 percent confidence level 
(p-value = 0.018).  All rows do not equal 100 percent due to rounding. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  11  Report Template Version  = 01-13-05 
FF II NN DD II NN GG SS
  
Table 5: Caller Experience with Receiving an Answer to Their 
Question as Quickly as Desired 
Percentage of Callers Reporting Receiving an 
Answer to their Question 
Type of Call Center 
As quickly as  Not as quickly 
Unsure 
desired  as desired 
Small Call Centers  77%  21%  2% 
Large Call Centers  88%  7%  4% 
1-800-MEDICARE  80%  16%  4% 
Source: Office of Inspector General analysis of caller surveys, 2004. 
We also captured comments from callers regarding multiple transfers 
and multiple calls to different telephone numbers. Nine percent of all 
sampled callers shared experiences such as referrals to other sources 
(e.g., telephone numbers) that did not resolve their issue or could not 
answer their question. As one caller stated, Who to call is a secret. It 
was a round robin to get to the right telephone number. During the 
week of our fieldwork, sampled callers made an average of 2.43 calls to 
call centers, ranging from 1 to 8. 
In addition to asking callers about 
Callers and call center managers both 
their experiences during their call 
placed a high priority on accuracy of 
to a Medicare-funded call center, 
answers, yet evidence suggests 
we also asked them about their 
oversight of accuracy may be inadequate  priorities for customer service. 
We identified three priority areas for telephone customer service based 
upon our review of the QCM process and discussions with CMS officials: 
(1) accuracy, (2) courtesy, and (3) amount of time caller spends on the 
telephone. While we recognize that all of these elements are important 
to the customer service experience, we asked both callers and call center 
managers to rank the three elements in order of importance. 
Overall, 67 percent of all callers during our week of fieldwork ranked 
accuracy as the highest priority. Forty-nine percent of all callers 
ranked their priorities as accuracy, courtesy, and time spent on the 
telephone, while 18 percent of all callers ranked their priorities as 
accuracy, time spent on the telephone, and courtesy. We believe it is 
OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  12  Report Template Version  = 01-13-05 
F I N D I N G S  
important to note that 24 percent of callers reported not receiving an 
answer to their question or all the information they needed. Both of 
these experiences are related to accuracy.   
Of the call center managers who ranked their customer service 
priorities,13 71 percent (37/52) ranked their priorities as accuracy, 
courtesy, and time spent on the telephone. The only other combination 
of priorities for call center managers, as reported by 29 percent (15/52), 
was courtesy, accuracy, and time spent on the telephone. 
Oversight of accuracy may be inadequate 
We accompanied a CMS review team on a performance evaluation in 
June 2004 and observed that the evaluation focused on validation of 
reported performance results (e.g., number of monitored calls, number 
of calls answered in less than 60 seconds, average speed of answer) and 
documented training activities.  However, the review team spent little 
time assessing the accuracy of answers given to callers.   
The review protocol requires reviewers to observe and listen to eight 
calls that the call center is monitoring as part of its QCM process. In 
practice, the review team observed, but did not evaluate the accuracy of 
four calls being scored. We inquired of the review team why they only 
observed the calls without evaluating the accuracy of the answers. A 
CMS review team member told us that evaluating the accuracy of the 
calls was not feasible since the review teams do not always include staff 
with the necessary expertise.   
Our fieldwork supported GAOs findings from a July 2004 GAO report14 
which found that the CMS performance evaluation criteria are not 
designed to verify that [customer service representatives] responses to 
providers are accurate.  In addition, a December 2004 GAO report15 
found that CMS and its contractors do not emphasize [customer service 
representatives] ability to answer questions accurately using [CMS-
approved guidance]. 
13 One call center manager refused to rank priorities, and two call center managers 
indicated that both accuracy and courtesy were the top priority, followed by the amount 
of time spent on the telephone. 
14 GAO-02-249 Communications with Physicians Can Be Improved, which can be accessed 
at http://www.gao.gov/new.items/d02249.pdf. 
15 GAO-04-669 Call Centers Need to Improve Responses to Policy-Oriented Questions from 
Providers, which can be accessed at http://www.gao.gov/new.items/d04669.pdf. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  13  Report Template Version  = 01-13-05 
FFF III NNN DDD III NNN GGG
 SSS  
Some call centers conducted quality assurance activities that exceeded 
CMS requirements, including activities that focused on accuracy 
Thirteen call center managers indicated that they had enhanced their 
QCM process by: (1) evaluating more than three calls per month per 
customer service representative; (2) tying QCM results to performance 
evaluations and raises; (3) having representatives evaluate recorded 
calls; and/or (4) having representatives participate in calibrations, 
which are meetings held across call centers to ensure all quality 
auditors score calls consistently. 
Thirteen other call center managers reported administering periodic 
knowledge tests to customer service representatives. Some call centers 
administered graded written tests completed by the representative. 
Other call centers had supervisors pose as callers and ask a question. 
The supervisor then assessed the accuracy and courtesy of the 
representatives response. 
Four call center managers reported that they conduct both enhanced 
QCM and knowledge testing. These four call centers were all large call 
centers. The sampled callers to these call centers experienced higher 
percentages of receiving an answer to their question and all the 
information they needed when compared to all other call centers. 
Sampled callers to these call centers also reported higher satisfaction 
with the customer service they received compared to all other call 
centers. However, the callers to these four call centers comprised a 
small subset of sampled calls; the results are not statistically 
significant, yet suggest the potential that these efforts may improve 
accuracy and satisfaction. 
When we asked call center managers about QCM in general, call center 
managers reported some positive aspects of the process, including: clear 
expectations for both quality auditors and customer service 
representatives, random selection of calls for monitoring, and 
identification of training needs for the representatives and the entire 
call center based on errors found in monitoring. In fact, 27 percent of 
call center managers (15/55) supported more frequent call monitoring, 
believing the minimum number of calls per month required by CMS for 
each representative did not give an adequate assessment of a 
representatives performance. 
OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  14  Report Template Version  = 01-13-05 
FF II NN DD II NN GG SS   
  R E C O M M E N D A T I O N S  

Medicare telephone customer service is an important source of 
information regarding Medicare benefits for beneficiaries and their 
representatives.  Their informational needs will undoubtedly increase 
as the significant program changes contained in the MMA are 
implemented. Furthermore, section 921 of the MMA specifically calls 
for prompt responses to inquiries from Medicare contractors, and for 
contractors to monitor the accuracy, consistency, and timeliness of the 
information they provide.  Although our inspection found that 
84 percent of callers asked to rate satisfaction reported they were 
satisfied overall with the customer service received, 44 percent of callers 
reported difficulty accessing information from call centers.  The 
response completeness and IVRs contributed to the reported difficulties.  
We recommend that CMS: 
o 	 Strengthen current oversight to place greater emphasis on 
completeness of responses, greater efficiency for the caller, and 
accuracy of answers given by customer service representatives. 
CMS could achieve this by:  including CMS staff with expertise in 
call handling on national review teams for performance 
evaluations, requiring call centers to conduct periodic knowledge 
testing with minimum passing scores, and/or increasing the 
number of calls monitored in the QCM process. Given that some 
call centers are currently engaged in conducting periodic 
knowledge testing and increased monitoring, CMS may want to 
determine the impact these practices may have on quality 
outcomes. 
o 	 Continue to seek ways to improve the national IVR system.  We 
note that subsequent to our fieldwork, CMS moved the option to 
speak with a customer service representative to the main menu, 
thereby decreasing the amount of time it required to arrive at this 
option from approximately 2 minutes to 50 seconds. 
AGENCY COMMENTS 
In its comments to the draft report, CMS expressed appreciation for the 
Office of Inspector Generals analysis and recommendations to help 
improve the benficiary inquiry call centers.  CMS stated it will continue 
with quality assurance activities, some of which we suggested in our 
recommendations, already underway at the 1-800-MEDICARE call 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  15  Report Template Version  = 01-13-05 
R E C O M M E N D A T I O N S  
center.  CMS described these activities in detail.  However, CMS does 
not plan to invest significant resources into reengineering quality 
assurance activities at the fee-for-service call centers because this 
workload will be integrated into the 1-800-MEDICARE operation. In 
addition, CMS stated that an expert in the field of IVR is being 
consulted to assist with improving and enhancing the system.  For 
CMSs complete comments, see page 17 of this report. 
OFFICE OF INSPECTOR GENERAL RESPONSE 
We appreciate CMSs comments to this report and note that CMS is 
taking action to address issues raised in this report.  We acknowledge 
that CMS revised its Medicare Contractor Beneficiary and Provider 
Communications Manual in June 2005.  Changes were made to the 
report to reflect technical comments received from CMS. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  16  Report Template Version  = 01-13-05 
  A G E N C Y   C O M M E N T S  

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  17  Report Template Version  = 01-13-05 
A G E N C Y   C O M M E N T S

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  18  Report Template Version  = 01-13-05 
A G E N C Y   C O M M E N T S

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  19  Report Template Version  = 01-13-05 
A G E N C Y   C O M M E N T S

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  20  Report Template Version  = 01-13-05 
A G E N C Y   C O M M E N T S

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  21  Report Template Version  = 01-13-05 
A G E N C Y   C O M M E N T S

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  22  Report Template Version  = 01-13-05 
A G E N C Y   C O M M E N T S

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  23  Report Template Version  = 01-13-05 
A P P E N D I X ~ A  
  A P P E N D I X   ~   A  
Confidence Intervals for Key Estimates and Chi-Square Tests 
Caller Experiences 
Weighted Chi-Square Test 
Finding 
Degrees of 
P-value 
Freedom 
Relationship between finding the interactive voice response easy to use and 
satisfaction 
Percent Satisfied  Percent Unsatisfied* 
Interactive voice response easy to 
89.14  10.86 
use/Don't remember  1  0.012 
Interactive voice response not 
72.99  27.01 
easy to use 
Relationship between receiving an answer to a question and all information 
needed, and satisfaction 
Percent Satisfied  Percent Unsatisfied* 
less than 
Received answer  95  5  1 
0.001 
Did not receive answer  51.91  48.09 
Relationship between receiving an answer to a question as quickly as desired and 
satisfaction 
Percent Satisfied  Percent Unsatisfied* 
less than 
Received answer quickly  92.06  7.94  1 
0.001 
Did not receive answer quickly  30.77  69.23 
*Includes sampled callers who reported that they were unsure or neither satisfied nor unsatisfied, in addition to 
sampled callers who reported that they were unsatisfied. 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  24  Report Template Version  = 01-13-05 
A P P E N D I X ~ A  
Caller Experiences 
Confidence 
Finding  Point Estimate  Interval                 
95 Percent 
Callers asked to rate satisfaction who were satisfied overall with 
84.5%  80.1% - 88.9% 
the customer service received 
Callers who reported difficulty accessing information from call 
43.7%  37.7% - 49.6% 
centers 
Callers encountering an interactive voice response who 
31.6%  25.9% - 37.4% 
reported that it was not easy to use 
Callers to 1-800-MEDICARE who reported that the interactive 
39.7%  30.1% - 49.4% 
voice response was not easy to use 
Callers who reported not receiving an answer to their question 
24.4%  18.9% - 29.9% 
or all the information that they needed 
Callers who reported not receiving an answer as quickly as they 
12.4%  8.3% - 16.4% 
desired 
Customer Service Priorities 
Confidence 
Finding  Point Estimate  Interval                 
95 Percent 
Callers who ranked priorities as:  1) accuracy, 2) courtesy, and 
48.7%  42.5% - 54.9% 
3) time spent on the telephone 
Callers who ranked priorities as:  1) accuracy,  2) time spent on 
18.1%  13.4% - 22.8% 
the telephone, and 3) courtesy 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  25  Report Template Version  = 01-13-05 
  A P P E N D I X   ~   B  
Profile of Surveyed Callers 
Small Call  Large Call  1-800-
Total 
Centers  Centers  MEDICARE 
Who called? 
  Beneficiary*  69  70  81  220/305 
  Beneficiarys family member  27  32  15  74/305 
  Other (e.g., provider or insurance office staff)  4  3  4  11/305 
Why did they call? 
   Medical bill/summary notice  56  56  19  131/305 
Benefit question  19  25  48  92/305 
   Other (e.g., address change, multiple issues)  25  24  33  82/305 
How did they know what phone number to call? 
   Printed on a notice I received  51  61  36  148/305 
   Someone gave me the number  14  21  8  43/305 
   Medicare & You handbook  5  5  19  29/305 
   Previous knowledge  3  8  2  13/305 
   1-800-MEDICARE advertisement  3  0  6  9/305 
   Other (e.g., Medicare card, telephone book)  24  10  29  63/305 
Did they try to get their question answered through 
any other source before calling? 
   No other source consulted  72  78  70  220/305 
   Medicare handbook or brochure  3  0  2  5/305 
   Internet Web site**  1  1  1  3/305 
   Friend, family member, counselor  1  0  1  2/305 
   Other (e.g., call to provider, secondary insurance)  23  26  26  75/305 
How many times did they call a Medicare-funded call 
center during the week? 
1-3  79  88  83  250/305 
4-8  21  17  17  55/305 
   Average number of calls  2.52  2.34  2.44  2.43 
How did callers describe CSR***/recording pace, 
volume, and clarity?
   CSR/recording spoke at the right pace  85  93  77  255/271 
   CSR/recording spoke at the right volume  84  93  82  259/271 
   CSR/recording spoke clearly  88  95  80  263/271 
How did callers rate their experience with the 
customer service received? 
Satisfied  70  88  64  222/271 
Unsatisfied  14  4  16  34/271 
   Neither or Unsure  8  4  3  15/271 
*These totals include four callers who called on behalf of themselves and for family members.  Two of these callers were in the

large call center groups, and two callers were in the 1-800-MEDICARE group. 

**One caller in the 1-800-MEDICARE group asked a friend, family member, or counselor in addition to accessing an Internet 

Web site. 

***Customer service representative (CSR).

  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  26  Report Template Version  = 01-13-05 
  A C K N O W L E D G M E N T S  

This report was prepared under the direction of Brian T. Pattison, 
Regional Inspector General for Evaluation and Inspections in the 
Kansas City regional office, and Gina C. Maree, Assistant Regional 
Inspector General.  Other principal Office of Evaluation and Inspections 
staff who contributed include: 
Tricia Fields, Team Leader 
Steven Milas, Program Analyst 
Linda Paddock, Program Analyst 
Elander Phillips, Program Analyst 
Perry Seaton, Program Analyst 
Deborah Walden, Program Analyst 
Tricia Davis, Director, Medicare & Medicaid Branch 
Technical Assistance  
Robert Gibbons, Program Analyst 
  OEI-07-04-00030  MEDICARE BENEFICIARY TELEPHONE CUSTOMER SERVICE  27