Social Security Administration: Additional Actions Needed in Ongoing Efforts to Improve 800-Number Service
Published: 2005-08-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Social Security Administration (SSA) at some point touches the life of nearly every American. Each day thousands of people contact SSA to file claims, update records, and request information from its 1,300 field offices, website, and national toll-free 800 number. Implemented nationwide in 1989, SSA's 800-number has become a principal contact point for millions of individuals seeking agency services. Congressional requesters asked GAO to review the quality of SSA's 800 number in terms of caller access and agent accuracy of response and courtesy.
section_title: What GAO Found
paragraphs: Despite making improvements to its 800-number service, SSA still has difficulty keeping pace with caller demand for agent assistance. In 2001, SSA upgraded its 800-number network so that all callers could either access its automated services or be routed to the next available agent at any site in the network--a feat not possible under the previous system. The new network also enhanced SSA's ability to monitor and manage call traffic, agent availability, and network operations in real-time to ensure the network's integrity and the consistent delivery of services. SSA also expanded its automated and agent-assisted services accessible through the 800-number network. However, SSA's expansion of its automated services to reduce agent call burden has not had its intended effect, as callers continue to show a strong preference for agent assistance. In fiscal year 2004, about 51 million of the more than 71 million callers requested to speak to an agent. However, 8.7 million, or 17 percent, of these calls did not get through to an agent--a 2 percentage point increase over the previous year. SSA has taken steps to help agents provide callers with accurate information and consistent services, but still has problems with agents assisting callers in line with agency policies and procedures. SSA's training curriculum provides agents with a comprehensive overview of SSA programs. Agents are also encouraged to use available on-the-job resources, including a customized computer application that helps agents provide consistent service and accurate responses. Nevertheless, from 2001 through 2003, SSA did not meet its 90 percent target for service accuracy--that is, agents' performance in handling non-payment related issues in accordance with agency requirements. Although SSA has taken several actions to help agents improve their performance, including mandating agent use of the computer application, it has not yet determined why agent compliance with agency policies continues to fall short. SSA trains and monitors agents for courtesy and conducts periodic customer satisfaction surveys, but does not routinely capture all customer complaints about alleged agent discourtesy. Agents receive training on developing their interviewing and interpersonal skills, and SSA monitors agents to determine whether or not they are providing courteous service to callers. SSA monitoring indicates that agent courtesy levels are high. SSA solicits limited customer feedback on agent courtesy in its annual surveys and compiles general ratings, but these surveys do not ask callers for the reasons behind the ratings. Callers to the 800 number do complain of agent discourtesy, but SSA does not routinely document and assess all complaints. Some call center staff told us that when they receive allegations of agent discourtesy, they typically apologize for the discourteous service and may proceed to assist the caller without recording the complaint. SSA has feedback mechanisms in place to capture caller complaints, but these mechanism do not do so in a manner that allows SSA to assess complaints and identify corrective actions needed.

section_title: Letter
paragraphs: The Social Security Administration (SSA) at some point touches the life of  nearly every American. Its services include issuing Social Security cards,  maintaining earnings records, and administering some of the largest  federal benefit programs—Old Age and Survivors Insurance and Disability  Insurance—commonly referred to as Social Security—and Supplemental  Security Income (SSI). Each day thousands of people contact SSA’s 1,300  field offices, Web site, and national toll-free 800 number (1-800-SSA-1213)  to file claims, report events to update records, and request information  about SSA’s programs. Since becoming available nationwide in 1989, SSA’s  800 number has become a principal contact point for individuals seeking  agency services. Callers to the 800 number may opt to use a self-service  menu of automated services or request to speak with an agent. In fiscal  year 2004, callers placed over 71 million calls to SSA’s 800 number.
SSA has made improvements and adjustments to its 800-number service in  response to customer feedback. However, some problems have persisted.  For example, SSA reported that millions of calls were encountering busy  signals or being terminated before being served. An SSA survey of 800- number callers in September 2000 found that callers wanted easier and  faster access to live service—that is, to speak with an agent without  wading through the lengthy list of automated services. Furthermore,  callers commented that SSA could improve its 800-number services by  having agents with expertise provide consistent information and exhibit  courteous behavior and helpful attitudes.
In this context, you asked us to review the quality of SSA’s 800 number in  terms of caller access, agent accuracy of response and courtesy. This  report discusses SSA’s efforts to (1) improve caller access to the 800  number; (2) ensure that SSA agents provide accurate responses and follow  SSA requirements; and (3) ensure that agents treat callers courteously.
To conduct our work, we reviewed SSA’s policies and procedures for   800-number agents and interviewed SSA officials to develop information  on SSA’s telephone systems and services, training offerings and  requirements, and the steps they take to ensure that their agents provide  acceptable service. Specifically, we interviewed and obtained  documentation from officials at SSA headquarters in Baltimore, Maryland;  regional officials in 3 of the 10 regions; and staff working in a total of six  call centers—two in each location visited—Birmingham, Alabama; Kansas  City, Missouri; and Richmond, California. The call centers we selected  varied in the volume of calls handled and included those that handled   800-number calls on a routine basis and centers that handled calls as  needed. We reviewed SSA performance data related to access, accuracy,  and courtesy and determined the data to be sufficiently reliable for the  purposes of this report. We conducted our work from September 2004  through July 2005 in accordance with generally accepted government  auditing standards. Appendix 1 provides a more detailed description of our  scope and methodology.
subsections: section_title: Background
paragraphs: Before implementing the nationwide 800-number service, SSA delivered  most of its services to the public face-to-face in an SSA field office. In  1989, SSA implemented a national, toll-free 800 number to better enable  individuals to request information on SSA programs or report events that  affect their own or someone else’s SSA records or payments. SSA set up  the 800-number service with the expectation that callers would ask basic  questions and conduct simple business transactions, such as reporting  address changes and scheduling field office appointments. When a call  came into the 800 number, it would be routed to a local SSA call center.  This strategy resulted in a high busy rate. Troubled by high busy-signal  rates, SSA in 1996 added a nationwide automated menu to the 800 number  that allowed callers to conduct a limited number of transactions without  speaking to an agent.
In 1997, we identified a number of conditions that limited the effectiveness  of SSA’s 800-number service. For one, callers often reached a busy signal  instead of the automated menu or an agent. In addition, the automated  menu offered only a limited number of services. To reach an agent, callers  were required to select a specific topic about which they wished to speak  to an agent so that the system could direct their call to an agent in a call  center with the requisite subject matter expertise. This routing strategy led  to some call centers being overwhelmed with calls. Also, because agents  could not transfer calls, callers sometimes were inconvenienced by having  to redial the 800 number to complete their business.
Since the introduction of its nationwide 800-number service, SSA has  worked to keep pace with the public’s growing demand for telephone  services and interests in conducting more complex transactions over the  telephone. Today, calls made to the 800 number are answered at   44 geographically dispersed locations. A call placed to the 800 number  may be answered by agents located in any one of SSA’s 36 teleservice  centers, 6 Program Service Centers; or at one of 2 components within  SSA’s Office of Central Operations. Figure 1 shows the locations of these  call centers within the 10 SSA regions.
SSA staffs its 36 teleservice centers with approximately 4,060 teleservice  representatives who answer incoming calls to the 800 number. In addition,  each of SSA’s six program service centers, which are co-located on  teleservice center sites, has designated specialists, called “SPIKES,” who  have been cross-trained to provide back-up support in answering   800-number calls during peak call volume periods. The SPIKE staff is  comprised of various technical staff in the program service centers whose  routine responsibilities include processing claims, mailing out notices,  managing SSA’s debt collection activities, and handling reports of non- receipt of checks and representative payee issues.
SSA employs a cadre of approximately 2,030 trained SPIKES in its six  program service centers. When the volume of calls is expected to exceed  the levels that teleservice representatives can handle, SSA activates  SPIKES, diverting them from their routine responsibilities to answer  incoming 800-number calls. These peak calling periods typically occur on  the first day of the week, the first week of the month, and the first 3  months of the year. In this report, we refer to teleservice representatives  and SPIKES as “agents” and to teleservice centers and program service  centers as “call centers.”
SSA’s Office of Telephone Services (OTS) plans, implements, operates,  and evaluates SSA telephone service to the public delivered by way of the  national 800 number and field offices. OTS plans and conducts studies,  pilots, and analyses of 800-number and field office telephone operations to  assess and improve the service. It also provides direct support to call  centers and field offices, including developing and communicating uniform  operating policies and procedures. OTS staff works closely with SSA’s  vendor that supplies and manages the network hardware, software, and  telephone equipment used to support the 800-number service. OTS also  manages the 800-number network operations, designs and administers call  routing plans, and monitors call handling and adjusts call routing to handle  emergency situations.
Full-time SSA agents spend much of their time answering calls. These  calls may cover a broad range of inquiries about SSA programs and  procedures. Figure 2 shows the 10 most frequent reasons for calls to the  800 number in fiscal year 2003.
Agents’ time off the phone, such as for staff meetings, training sessions, or  annual leave must be scheduled months in advance so that the network  operations may continue without interruption.
SSA sets goals for telephone access and agent services and measures  performance in these areas. In recent years, to measure access, SSA  calculated the number of calls handled, the number of calls that reach the  800 number on their first attempt, and the number of calls that reach an  agent within 5 minutes of selecting the option to speak with an agent. In  fiscal year 2005, SSA replaced these measures with two new access  performance measures—the average speed of answer and the agent busy  rate—consistent with standards in the telecommunications industry. SSA  also expects agents to adhere to agency guidance and procedures and sets  standards and measures agent accuracy (i.e., compliance with SSA’s  requirements when serving callers) and agent courtesy. The Office of  Quality Assurance and Performance Assessment (OQA) measures the  accuracy of information agents provide callers by listening in daily to a  statistical random sample of calls handled by agents nationwide. OQA  assesses accuracy based on whether agents adhered to SSA requirements  when responding to callers’ inquiries. As shown in table 1 agents are  expected to provide callers a broad range of services. OQA also  periodically surveys 800-number callers to assess, among other things,  callers’ perception of agent courtesy.
section_title: SSA Has Improved Overall Access to the 800 Number, but Many Calls Seeking Agent Assistance Do Not Get Through
paragraphs: Despite making improvements to its 800-number systems, SSA still has  difficulty keeping pace with caller demand for agent assistance. Since  2001, SSA has made improvements to its telephone systems, management,  and services to improve caller access to the 800-number network.  Specifically, the new enterprise-wide network improved incoming call  routing and network capacity; enhanced SSA’s ability to manage network  operations, forecast call volumes, and set staffing levels; and expanded  automated and agent services. However, callers continue to demonstrate a  preference for speaking with an agent over using the automated service  menus. In fiscal year 2004, about 51 million callers requested to speak to  an agent. Of these calls, 8.7 million, or 17 percent, of these calls did not get  through to an agent—a 2 percent increase over the previous year.
subsections: section_title: SSA Made Numerous Upgrades to Its 800- Number Network
paragraphs: SSA upgraded the network to help overcome past access problems. One  major upgrade was the replacement of the geographically based routing  system with a nationwide routing system capable of distributing calls to  any agent within the network. This change gave SSA the ability to monitor  call traffic and agent availability in real time at each call center and receive  “cradle to grave” management information on a call’s movement from the  time the caller dials the 800 number until the call is terminated. The  network also effectively eliminated the busy signal that callers  encountered when using the older system. The new system accepts all  calls made to the 800-number network and provides callers with a broad  range of automated services. Calls seeking agent assistance are distributed  to 1 of SSA’s 44 answering sites.
When callers dial the 800 number, the network provides a series of  prompts to direct them to the desired services. The network uses recorded  announcements and pre-set menu prompts to separate callers according to  language preference (i.e., English or Spanish) or type of telephone service  (i.e., touchtone or rotary dial). The network uses a digitized voice to read  menu selections to the caller and responds to caller-entered touch-tone  digits. The caller’s selection can invoke a number of options, such as  playing a recorded announcement (e.g., on cost-of-living adjustments) or  transferring a call to an agent. SSA provides callers with an extensive  menu of available automated services before offering them the option of  acquiring agent assistance. SSA told us that the menus were set up this  way to offer callers an opportunity to conduct their business using  automated services before forwarding their calls to agents.
When a caller indicates a preference for agent assistance, the network  determines the optimum destination for the call. It reviews among other  factors agent availability, the number of calls in queue, and the minimum  expected delay. If all agents are busy and call queues are filled to capacity,  the network delivers an agent busy message to callers, advising them that  heavy call volume prohibits the transfer of their call to an agent and  encouraging them to call back during periods of typically lower call  volumes.
A call placed in agent queue remains queued until an agent becomes  available. The network applies treatments to calls waiting in agent queue,  such as announcements promoting the use of SSA’s Web site. According to  SSA, if the wait time in an agent’s queue exceeds 15 minutes, the call is re- routed to another agent and given priority over other incoming calls. The  network continually tracks the status of each call until the caller  disconnects. Although the network was designed to hold one call per  agent in queue, the vendor told us that it typically holds up to 1.65 calls in  queue per agent.
SSA and the vendor have taken measures to ensure the integrity of  network-generated data and the continuous operation of the network.  Both SSA and the vendor conduct ongoing tests of the accuracy and  completeness of the network-generated data on which so much of SSA’s  800-number related performance measurement, management decisions,  and staffing levels depend. The vendor told us that redundancy was built  into the network to ensure that the failure of any one component only  affected existing calls. For example, if one component fails, the network  automatically employs a backup execution path to bypass the problem  location and reroutes calls to one of the remaining call centers. According  to the vendor, the redundancy built into the 800-number network and the  geographical dispersion of its redundant functions would make a complete  system outage almost unimaginable. Vendor staff told us that the local  outages that occur on occasion are mainly caused by loss of network  facilities, extended local power failures, or hardware issues. SSA and the  vendor maintain back up databases critical to network operations.
SSA takes several additional steps to help ensure that callers can access  800-number services. SSA network operations staff frequently calls the  800-number network to test the integrity of the main menu scripts and the  routing of calls to both automated and agent services. They evaluate calls  for proper routing through the option choices; proper functionality of the  automated scripts; proper functionality of routing to agents; proper  routing to agents and agent queues; and the quality and clarity of the  connection. Call centers also have systems administrators who monitor  the performance of the equipment used on the premises and notify  headquarters when any anomalies appear. System administrators are  responsible for keeping the phones and headsets working, troubleshooting  problems with desktop applications, monitoring computers, printers, or  management information data. If the administrators notice any problems,  they are responsible for notifying headquarters so that the vendor can  dispatch a technician to initiate repair.
section_title: SSA Manages Call Traffic and Forecasts Future Call Demand and Staffing Needs
paragraphs: SSA takes advantage of the wealth of management information at its  disposal to monitor ongoing network operations and plan for the future.  SSA forecasts call volumes and schedules the appropriate number of  agents in accordance with anticipated demand based on historical data.  These forecasts allow SSA to group days into specific levels depending on  the anticipated volume of calls. For example, the busiest days—”Level 1”  days—require the greatest number of SPIKES to be activated to answer  phones. SSA sets and tracks SPIKE commitments to help ensure that  enough SPIKES will be available networkwide to answer the volume of  incoming calls. Depending on network conditions, managers may make  adjustments to the number of available agents and the routing of calls to  align available 800-number resources with caller demand. SSA adjusted its  call volume forecast downward 5 times each in fiscal years 2003 and   2004, allowing SPIKES scheduled to answer 800-number calls to return to  their other assigned duties.
SSA uses real-time data to monitor call traffic, caller activity, and system  performance. SSA can use these data to track overall incoming calls and  information on automation or determine whether calls were routed to an  SSA call center or to a busy message. SSA monitors such 800-number  network statistics as calls made to the network, calls offered to agents,  agent staff levels, average speed of answer, and agent busy rate. Staff also  monitors cable and national news for events, such as inclement weather,  news stories on Social Security, or homeland security events to determine  what impact they may have on projected 800-number call volumes.  Furthermore, SSA monitors caller usage of the automated menus and  reshuffles automated options to keep the most popular options first.
SSA performs limited checks of the network generated data. Upon  receiving the data electronically, SSA runs the data through a multistep  automated procedure that backs up the data and converts it to a readable  format. As part of this process, SSA checks each record to ensure that all  area codes are good, all phone numbers are properly formatted, and all  listed phone numbers originate in the 800-number network. The vendor  also generates separate reports on automated services and agents. SSA  reviews the reports and compares the results with historical trends.  Although SSA has no additional means of verifying the reliability of the  vendor-provided data or the results that appear in report field outputs,  both SSA and the vendor maintain that these data are accurate, and the  vendor states that SSA has the source data it needs to assess network  performance.
section_title: SSA Expanded Its Automated and Agent- Assisted Services, but 17 Percent of Calls Seeking Agent Assistance Do Not Get Through
paragraphs: Since the inception of the nationwide 800 number and the later  introduction of limited 24-hour automated services, SSA has continually  improved the quality and quantity of services available to callers. In   1996, SSA introduced voice-recognition applications and added an option  allowing callers to replace their Medicare card by phone. In 1998, SSA  implemented five new automated service options to handle inquiries  surrounding the increased number of Social Security statement mailings.  By 2002, SSA had made the full range of automated services available in  the Spanish language.
Callers may access the automated services at any time in English or  Spanish to obtain services, information, or forms. Table 2 lists the services  available through the 800-number automated menus.
SSA has adopted the telephone industry “best practice” of taking care of  all of the caller’s business during the initial contact. Agents have been  trained to answer a wide range of inquiries and have the capacity to  transfer calls they cannot handle to others who handle these calls. For  example, in 1998, SSA began allowing callers to file claims for retirement  and survivors’ benefits immediately through the 800 number, eliminating  the need for the caller to leave a message and wait for another SSA agent  to return the call. In 1999, SSA gave agents access to a computer-based  application to assist them in handling telephone calls more efficiently. In  2002, SSA provided callers the option of having their call routed to a  designated group of bilingual agents. SSA also extended the hours of agent  availability nationwide. Agents are now available weekdays from 7 a.m. to  7 p.m. in each time zone. In addition, SSA provides unadvertised agent  service for extended hours on weekday nights and weekends. SSA also  provides agent service for the hearing impaired through a separate toll- free number.
In following SSA’s instructions to handle all of the caller’s business needs,  agents may be performing tasks that limit their availability to answer calls.   During site visits, we observed agents who filled out forms by hand,  retrieved printouts, placed the mailings in an envelope, addressed the  envelopes by hand, and put the envelope in the mail slot, while the caller  remained on hold.  While these steps may help give callers the assurance  that their business is being completed, such manual tasks are time- consuming and potentially limit the number of calls that agents can  handle.
Although the number of calls placed to the 800 number has increased  slightly since fiscal year 2002 and SSA has expanded services available  through automation, the agency continues to have difficulty keeping pace  with caller demand for live agent assistance. Figure 3 shows the calls  made to the 800 number since fiscal year 2002 when SSA’s most recent  telephone network upgrade was fully implemented. The proportion of  calls to the 800 number indicating a preference for agent assistance has  been relatively consistent, whereas SSA had hoped that the introduction of  automated services would divert calls away from agents to the less costly,  self-service automated system.  Such a reduction would be consistent with  the call center industry trend toward self-service with minimal agent  intervention.  However, agents continue to answer the majority of calls,  including some calls that, according to agents, could easily be handled  through automation. The percentage of calls seeking agent assistance but  not getting through declined from 22.7 percent in fiscal year 2002 to   15.2 percent in 2003, but rose 2 percent in fiscal year 2004. Specifically, as  figure 4 shows, 8.7 million (or 17.2 percent) of the 51 million calls seeking  agent assistance in fiscal year 2004 did not get through. About half of these  calls encountered a busy message and the other half abandoned the call  while waiting in queue.  Managers of private call centers do not place a lot  of importance on call abandonment rates for several reasons, including  their belief that callers terminate calls to visit the Web site.  Some callers  that request SSA agent assistance may be able to satisfy their needs  through the automated menu or Web site. However, if callers’ business  require agent assistance, they will not be able satisfy their needs if they  unable to get reach an agent.
SSA offers a variety of possible reasons why callers abandon their calls  after being placed in queue for an agent, one being that customers simply  do not want to continue waiting any longer before having an opportunity  to speak to an agent. SSA has several initiatives underway to reduce the  number of abandoned calls in queue, including a call-back service, which  will provide callers kept in queue beyond a certain threshold with an  opportunity to enter their telephone number and select a contact time so  that an agent can call them back. While providing convenience to callers  and potentially using any agent “down” time more efficiently, a call back  option also has the potential to increase agent workload.
Since 2002, SSA’s 800-number automated menus have received  progressively higher call volumes but handled fewer calls to completion.  In addition, as shown in figure 5, the number of calls being abandoned  without completing a transaction in the automated menus has steadily  increased, culminating in fiscal year 2004, when nearly half of calls to  automation were abandoned.
Although SSA offers a number of possible reasons, it is unable to say with  any degree of certainty why calls continue to be abandoned. In the past,  SSA has conducted follow-up caller surveys to ask callers what had  prompted them to abandon the automated services. The primary reason  that callers gave for hanging up after an initial selection of an automated  service was their desire to speak to an agent. According to SSA, many  callers simply desire the security of human contact when leaving personal  information that is required to transact business. SSA has now eliminated  the need for callers to redial; callers may now have their calls transferred  from automated services to agent queue. However, this option will likely  increase agents’ call burden. SSA intends to make its automated menu  selections more accessible by introducing a speech-enabled main menu  that would allow callers to simply speak their needs in response to  directed questions. For example, rather than listening to a list of options,  callers will be able to use their voice to narrow down available options and  find the ones relevant to the services they seek. SSA plans to implement  this feature nationwide later in this fiscal year.  SSA also redesigned its  Web site in 2003 to improve its accessibility and usability in the hope of  relieving the burden on the 800 number. The Web site now attracts over   30 million visitors a year, which SSA says has reduced the demand for  direct service from 800-number and field office agents. SSA’s customer  satisfaction surveys from 2002 and 2003 show that the percentage of the  survey respondents who said they would likely use the 800 number the  next time they contacted SSA decreased from 75 to 61 percent. In contrast,  the percentage of respondents who reported they were likely to use the  Internet or email to contact SSA increased by 2 percent and the  percentage of those who said that they would likely call or visit a field  office increased by 10 percent.
section_title: SSA Trains and Provides Agents on- the-Job Resources, but Agents Have Not Met SSA’s Standard for Accuracy of Assistance
paragraphs: SSA has taken steps to help agents provide callers accurate information  and comply with agency requirements, but still has problems with agents  meeting its standards for accurate service. SSA provides agents with  comprehensive training and equips them with on-the-job resources to help  them provide accurate and consistent service. In addition, SSA monitors  agents’ calls and compiles agencywide assessments of agent accuracy in  handling calls and identifies agent training needs. SSA’s own monitoring  assessments for 1998 through 2003 found that the agency generally met its  standard for agent accuracy in handling issues that had the potential to  affect individuals’ benefit payments, but not its standard for “service  accuracy,” handling issues that did not have the potential to affect  benefits. SSA’s overall performance for “service accuracy” for fiscal year  2003 was 85.1 percent; below SSA’s 90 percent target. According to SSA’s  assessment, agents’ failure to obtain the required identifying pieces of  information from callers to verify their identity before accessing and  disclosing information was the most frequently committed service error. In  fiscal year 2003, this error alone accounted for 28 percent of all service  errors that SSA identified. SSA has taken several actions to help agents  improve their performance, but these actions have not resulted in  sustained improvements in service accuracy.
subsections: section_title: SSA Trains Agents and Equips Them with Resources to Properly Handle Calls
paragraphs: SSA provides agents with comprehensive training to enable them to offer  callers a broad range of services and to complete callers’ business on  initial contact. The basic training curriculum is comprised of formal  course work to teach agents about the agency’s programs, policies, and  procedures, including rules for disclosing information to and accepting  reports from callers; how to access, interpret, and enter data into SSA  computer systems and databases; and how to query and interpret SSA  records. As part of their basic training, agents take frequent tests, conduct  mock interviews, observe experienced agents handling calls, and answer  calls. The basic training curriculum for full-time agents at the call centers  we visited ranged from 11 to 16 weeks. In addition, call center officials  told us that they taught a modified 11- to 12-week course to back-up agents  to augment their existing technical skills. In addition, officials told us that  they supplemented the basic training with regional and call center training  offerings, such as new employee orientation, diversity training, and public  service training.
After agents complete basic training, regions and call centers follow their  own established practices to help agents transition to handling calls on  their own. At the sites we visited, agents were mentored or closely  supervised during a transitional period. For example, some call centers  assigned a personal mentor to sit and observe agents handling calls and to  provide prompt assistance, as needed. After spending a number of weeks  with a mentor, agents are evaluated to determine their readiness to handle  calls on their own. As another transitional step, one call center placed  agents in a training unit that had a higher supervisor-to-agent ratio to allow  closer supervision and monitoring of agents’ work. Floor support staff in  one training unit said that, in addition to providing technical assistance,  they review the accuracy of agents’ data entries for events, such as direct  deposit requests and death reports. Based on an agent’s proficiency, floor- support staff may review agents’ work to provide daily feedback or review  their work less frequently as agents demonstrate proficiency.
Agents may receive subsequent training in a variety of ways. For example,  training can occur during the 3-hour allotments reserved for monthly staff  meetings. Call center staff and officials told us that these meetings were  used as a forum to provide agents information on emerging issues such as  national and regional initiatives and changes in operating procedures, as  well as feedback on the call center’s performance. During the workday,  supervisors may provide agents with important information that agents  need to know, such as generic responses to calls triggered by current  media reports on Social Security solvency. We were told that agents also  receive voluminous intra-agency communications for which they may be  allotted 15 minutes at the end of each workday to read. We were also told  that supervisors and floor-support staff use various strategies to ensure  that agents are aware of the most important changes. Call center managers  and supervisors told us, that if needed, more time maybe requested for  agents to be off the telephones to receive additional training, such as  hands-on computer training.
To assist agents in providing callers with accurate and consistent services,  SSA provides agents with the Customer Help and Information Program  (CHIP)—a customized online computer application for providing services  to 800-number callers. CHIP helps agents navigate the comprehensive set  of requirements and guidance for SSA programs and directs agents in the  actions they should take to accurately complete callers’ business on initial  contact. For example, if an agent enters an address change for individuals  receiving Supplemental Security Income (SSI) benefits, CHIP displays  screens prompting the agent to ask callers a series of questions about  changes in living arrangements—events that may lead to an increase or  decrease in SSI benefits. As another resource, the call centers we visited  made more experienced staff available to help agents handle more  complex or technical calls. Officials told us that such floor support was  customary at call centers agencywide.
section_title: SSA Monitors Agents to Assess Performance and to Identify Training Needs
paragraphs: SSA monitors agents’ handling of 800-number calls for payment accuracy  and service accuracy. SSA assesses agent performance for payment  accuracy in cases where agents’ responses on such matters as eligibility,  filing of claims, or reportable events could potentially affect an individual’s  eligibility or benefits. SSA also assesses agents’ performance for service  accuracy to determine whether or not the services they provide  correspond with SSA policies and procedures. When assessing service  accuracy, SSA considers whether agents provided accurate information as  well as performed all other related actions that the agency requires. Some  of these actions are required as a matter of convenience to callers or to  avoid the potential need for follow-up contact.
SSA conducts random, remote monitoring of agents handling calls for  various purposes. OQA is responsible for two types of monitoring. First,  OQA monitors a statistical national sample of calls handled by agents  throughout the year to develop both agencywide and regional assessments  of agent performance. This type of monitoring serves as SSA’s means of  assessing agent payment accuracy and service accuracy. OQA officials told  us that such monitoring had the capacity to reveal issues that needed to be  addressed at the agency level, such as pinpointing areas needing policy  clarification. However, the responsibility for agent performance, including  improving performance to meet agency targets, rests with the various  regions and individual call centers. Second, if requested by regional  officials, OQA occasionally monitors a small number of calls handled by  individual call centers and visits the call centers to brief managers and  agents on its findings. Call center staff also randomly monitor calls  handled by their call center for payment accuracy and service accuracy  and to identify training needs for their agents. SSA does not specify the  number of calls that should be monitored for this purpose. Call center  officials told us that the number of calls they monitored do not provide a  statistically valid assessment of their center’s performance.
Designated call center personnel also monitor individual agents to provide  agents individualized feedback on their telephone performance. Monitors  may point out positive aspects of agents’ performance as well as suggest  additional training. Agents are given advance notice of when monitoring  will occur and are allowed to choose whether to have monitors sit with  them or to have monitors listen in from a remote location. For full-time  agents, SSA guidance recommends monitoring as many as five calls per  month for agents with more than 1 year of experience and unlimited calls  for agents with less than 1 year. Officials told us that agents are given  timely feedback on assessments of their overall performance. Some  officials also said that when monitors observe agents making an error,  they may interrupt the call to instruct the agent on the correct procedure.
section_title: Agents Have Not Met SSA’s Standard for Accuracy of Assistance
paragraphs: Although SSA takes a number of actions to help agents provide callers  accurate information in accordance with agency policies and procedures,  agents still have problems meeting SSA’s standard for service accuracy. As  shown in figure 6, from fiscal year 1998 through fiscal year 2003, SSA  generally met its 95 percent target for payment accuracy—having agents  correctly handle inquiries involving eligibility and benefit payment  issues—but not its 90 percent target for service accuracy—having agents  serve calls related to nonpayment –related issues according to agency  requirements.
SSA reported that its overall performance for payment accuracy in fiscal  year 2003 was 95.9 percent, and the performance for each of its 10 regions  was similar. However, SSA reported its overall performance for service  accuracy in fiscal year 2003 was 85.1 percent. Based on OQA’s assessment,  as few as four regions may have met the 90 percent service accuracy target  in 2003. As shown in figure 7, for fiscal years 2001 through 2003, almost  all regions had problems consistently meeting SSA’s established target for  service accuracy.
OQA identified 63 types of required actions that agents failed to take in  fiscal year 2003 that led SSA to miss its service accuracy target. Agents’  failure to take these required actions resulted in service errors. As shown  in table 3, the most frequent error stemmed from agents’ inadequate  protection of individuals’ personal information. SSA protects individuals’  privacy by limiting disclosure of the personal information in its records to  individuals for whom the agency maintains the records and to others  authorized. Agents committed an error each time they failed to collect the  requisite six identifying pieces of information to verify a caller’s identity  before accessing or disclosing information from SSA records (i.e.,  improper handling of access and disclosure).
Managers at the sites we visited have taken actions to reduce the number  of service errors, particularly access and disclosure errors. For example,  some call centers provided CHIP refresher training, designed desk aids  reminding agents of the steps for proper disclosure, placed “hot pink”  sheets detailing the service errors on the desks of agents who commit  them, or established a “CHIP doctor” to provide agents with technical  assistance to help navigate the CHIP computer application. However, the  effectiveness of these actions to improve service accuracy for agents  within the respective call center is unknown because the monitoring that  occurs at the call center level does not provide a statistically valid  measurement to make such an assessment.
OQA has reported that the lower service accuracy rate primarily stemmed  from agents’ failure to follow SSA’s requirements when asking callers to  verify their identities. Assuming that such “access and disclosure” failures  could be cleared up through the use of the CHIP application, SSA  mandated its use in November of 2001. The service accuracy rate  subsequently improved for fiscal year 2002, but dropped the next fiscal  year because, according to OQA, agents did not make optimal use of CHIP.  OQA recomputed SSA’s service accuracy rate without the access and  disclosure error for comparison purposes and reported that it would have  increased from 85.1 to 89.2 percent in fiscal year 2003. SSA has not  determined why agents fail to follow agency procedures when handling  some calls, resulting in service errors.
section_title: SSA Conducts Training, Monitoring, and Customer Surveys but Lacks a Uniform System for Assessing Agent Courtesy
paragraphs: SSA uses training, call monitoring, and surveys to ensure that agents  deliver courteous service, but does not routinely document or analyze all  incidents of discourtesy or caller complaints. As part of its comprehensive,  multiweek training curriculum, SSA teaches agents the interviewing and  interpersonal skills they need to provide courteous service. It also  determines through monitoring whether agents are being courteous. Based  on its monitoring results from 2001 through 2003, SSA reported that it  found agents to be courteous to callers over 99 percent of the time. SSA  also measures caller satisfaction with agent courtesy as part of its annual  800-number customer satisfaction survey. In 2004, 91 percent of  respondents rated agent courtesy as good, very good, or excellent;   5 percent rated agent courtesy as fair, and 4 percent rated it poor or very  poor. While SSA uses training, monitoring and customer surveys to ensure  courtesy, it does not have a uniform system for analyzing incidents and  complaints of discourtesy. Call center staff told us that they typically  apologize to callers and offer to provide the desired assistance whenever  callers lodge complaints by phone. Moreover, they may not record the  complaint or attempt to capture and assess information on the nature of  complaints.  Customer service studies highlight the importance of paying  attention to complaints and the benefits of having a good management  complaint system.
subsections: section_title: SSA Relies on Training, Monitoring, and Customer Surveys to Ensure Agent Courtesy
paragraphs: As part of its comprehensive, multiweek training curriculum, SSA teaches  interviewing and interpersonal skills to help agents serve callers in a  professional and courteous manner. The training includes instruction on  how to establish rapport with callers, how to obtain information necessary  to accurately serve callers’ needs, and how to end calls on a positive note.  As a courtesy measure, agents are instructed to allow callers to end the  call. Agents also receive training on how to respond to angry, loud, or  abusive callers, including how to calm such callers, and how to continue  serving them or to transfer those calls to supervisors.
SSA also uses its call monitoring process to oversee courtesy levels and  has procedures for immediate intervention to remedy any observed  problem. OQA procedures call for monitors to immediately inform  management of a discourteous incident, prepare a written report for the  agent’s call center manager, and retain a copy of the report in the event  that a disciplinary action is taken against the agent. Call center managers  who become aware of discourtesy allegations or observe agent  discourtesy are required to follow similar procedures. They are required to  discuss any incident with the agent and consider a progressive range of  disciplinary actions from issuing a reprimand to terminating an agent’s  employment.
OQA officials told us that formal monitoring is time-consuming work. As a  result, OQA said that over the years, it reduced the sample size of the  monitored calls due to resource constraints. Regional and call center  management expressed varied opinions as to whether the reduction in the  number of monitored calls was an obstacle to identifying agent  discourtesy. One call center manager told us that discourtesy was more  likely to be observed by managers and supervisors patrolling work areas  than through formal monitoring. On the other hand, one regional official  noted that additional unannounced monitoring would be a more effective  way of catching agent rudeness.
Some of the managers and officials with whom we met, however, told us  that they believed courtesy levels were very high and not a problem.  According to the agency’s call monitoring records, SSA agents have  performed at consistently high rates with regard to courtesy. For fiscal  year 2003, OQA determined that based on 4,384 calls, agents had been  courteous to callers 99.9 percent of the time.  It reached similar  conclusions from its 2001 and 2002 monitoring.
SSA also relies on its annual survey of callers to assess and ensure agent  courtesy. Callers who have used agent services have been asked, among  other questions, to rate agent courtesy on a 6-point scale. The 2004 survey  showed that 91 percent of the callers rated agent courtesy as being good,  very good, or excellent; 5 percent rated it as being fair; and 4 percent rated  it as being poor or very poor.  These rates were about the same as those  reported for the 2001 though 2003 surveys.
Other call centers may use telephone or online surveys to obtain feedback  from customers, although the actual administration of the surveys may  vary.  For example, one organization conducts telephone surveys using  voice capture software to record customer responses.  At the beginning of  a call, the survey system randomly selects participants and asks them to  participate in a 2-3 minute survey after they complete their call. Another  organization conducts online surveys, sending a survey to selected  customers via email.  Nonetheless, these organizations seek to obtain  customers’ views on their organization’s performance.
section_title: SSA Lacks a Uniform System for Assessing Agent Courtesy
paragraphs: SSA monitors calls and receive feedback from customers, but it does not  systematically gather and assess this information to identify courtesy  problems, such as particular problem locations or persistent patterns or  trends. SSA agents handled an average of 40.9 million calls each year from  2001 through 2003. Even if agents were courteous 99.9 percent of the time  as OQA reported, for fiscal year 2003 that would still leave nearly 60,000  calls in which the agents may have been discourteous. However, because  SSA does not routinely analyze the details of agent discourtesy observed  through monitoring, it has no way of determining the circumstances or  lessons learned from monitored calls.
Studies conducted on customer service have shown that building  relationships with customers and a having first-rate management  complaint system are critical to maintaining good customer relations. One  study in particular noted that paying attention to customer complaints,  regardless how minor they may be and addressing them quickly and  completely helps satisfy customers and build trusting relationships.
Similarly, people who contact their government agencies want to be heard  and expect courteous and respectful treatment. It is therefore important  for government employees to distinguish what their customers want and  to take actions to ensure that their customers are satisfied. The study also  noted that no matter how good the service or product is, occasions will  invariably arise that result in customer complaints. However, it is  important that when criticisms are voiced, that they are systematically and  promptly addressed.  A good management complaint system can provide  data and information on complaints that can be compiled and analyzed to  give insight into where problems are recurring and what needs to be done  to fix them or prevent them from happening in the future. A good  complaint system also facilitates the filing of complaints using simple, yet  comprehensive complaint forms.
SSA’s 800-number customer satisfaction surveys are one means of  gathering feedback from callers on agent courtesy. However, the survey  does not ask why some respondents rate agent courtesy as poor. In  addition, the agency does not routinely collect or analyze all caller  complaints placed through the 800 number. Our visits to call centers found  variation in how they handled such calls. When customers call the   800 number to report agent-related complaints, SSA guidance requires  agents to refer calls to supervisors or floor-support staff. However, SSA  does not provide guidance for how those receiving referrals should handle  them. We were frequently told that call center staff receiving these calls  typically apologize for the other agents’ rudeness and offer to provide  service to the caller. SSA provides call center staff a form to document  800-number service complaints, including agent lack of courtesy.  However, SSA has not provided them agencywide guidance on  documenting complaints or the type of information they should record to  allow SSA to identify service issues or trends. We were given a variety of  reasons why call center staff may not document agent-related complaints.  One call center official told us that his site allowed agents to exercise  judgment in deciding which complaints they documented. Some agents,  supervisors, and technical staff told us they were unaware of procedures  for handling such complaints while others believed callers needed to  provide sufficient information such as the offending agent’s name or call  center location to lodge a formal complaint. It was our observation that  800-number agents may not provide their full name or mention their call  center location when answering a call.
SSA responds differently to customer-reported complaints sent to agency  offices than to complaints registered on its Web site. Specifically, regional  and call center officials said that when warranted they would attempt to  identify the agent, investigate the merits of the complaints reported to  their offices, and initiate disciplinary actions. Headquarters staff who  receive complaints through the agency’s Web site told us that they  routinely send customers a letter of apology, but have no one designated  within SSA to whom to forward and or resolve complaints. Although the  Web site has an Intranet-based form ostensibly designed to capture  complaint information, it does not ask for specific information such as the  nature of the alleged act of discourtesy and the date and time it occurred.  By not systematically collecting and analyzing information on alleged  agent discourtesy, SSA is unable to identify service issues that may  warrant corrective actions.
section_title: Conclusions
paragraphs: SSA’s toll-free phone service is an important resource for the thousands of  people who call the number daily, and the steps the agency has taken in  recent years demonstrate a commitment to quality service. The agency’s  upgraded telephone system along with its expanded menu options and  agent-assisted services has improved access in many respects by giving  customers more services at their convenience. In addition, the agency has  taken steps to help ensure that callers receive more accurate and  courteous service.
Even with good service, however, there is room for improvement.  Improvements to the 800-number network have not necessarily ensured  that callers receive the help they seek, given the number of calls not  reaching an agent. This may not be a mounting problem if callers follow  the general trend in the call center industry toward automation and self- service as they grow more comfortable with these options. However, the  impending increase in the size of the retiree and disability populations,  and anticipated changes to the Social Security system suggest that SSA  may continue to experience a substantial proportion of callers who  request agent assistance. Measures to improve customer access to agents  may therefore be needed. In addition, SSA’s many benefit programs will  continue to generate some complex questions that require agent  assistance. Currently, the prevalence of service accuracy errors diminishes  the quality of service that callers receive when they do reach an agent  through the 800 number. Finally, although SSA’s estimates show instances  of agent discourtesy to be rare among all calls, such instances could  nonetheless affect tens of thousands of callers. Because SSA does not  routinely capture information on all customer complaints about  discourtesy, however, it loses the ability to assess the severity of the  problem and misses opportunities to better understand caller needs, solve  unanticipated problems, and retain the good will of the public.
section_title: Recommendations for Executive Action
paragraphs: To improve the quality of the 800-number telephone service, we  recommend that the Commissioner of the Social Security Administration  take the following three steps:  Identify cost-effective ways that will help ensure that more calls  seeking agent assistance get through to agents, such as streamlining  the call-handling process, automating some mailings that agents now  do by hand, or increasing number of agents available to take calls.
Conduct a comprehensive analysis of the source of service errors. For  example, the agency might consider holding agent focus groups to gain  insight into why agents tend to fail to comply with certain  requirements. The agency could get agents’ views on the effectiveness  of CHIP in helping them meet agency requirements.
Establish procedures for documenting and assessing customer- reported complaints. In doing so, the agency should determine the  types of information it needs to assess customers’ concerns and to  provide the agency a means to identify and address service issues.
section_title: Agency Comments and Our Evaluation
paragraphs: We obtained written comments on a draft of this report from the  Commissioner of SSA. In its comments, SSA said it was pleased that our  report reflected the agency’s commitment to providing high-quality   800-number telephone service that meets the needs and expectations of its  customers. SSA agreed with our recommendation to identify cost-effective  ways to increase agent availability to handle 800-number calls and  described several planned initiatives to improve agent productivity and to  expand automated services. SSA also agreed with our recommendation to  conduct a comprehensive analysis of the source of agent service errors.  Accordingly, SSA said it would convene a workgroup to obtain feedback  on the source of agent service errors and make recommendations as  appropriate to improve the agency service accuracy level.
SSA disagreed with our recommendation to establish procedures for  documenting and assessing customer complaints. SSA said that its findings  that agent courtesy levels are consistently high demonstrate that its  present approach to ensuring agent courtesy—which combines training,  monitoring, and customer surveys—is working. Moreover, SSA said that  based on its experience with prior initiatives, a nationwide reporting  system would require heavy resource expenditures and be cost prohibitive  given current budget constraints. Furthermore, SSA stated that any use of  agent resources to document complaints would be counterproductive to  improving caller access to agent services.
While we agree that agent courtesy levels are high and state this in the  report, given the sheer volume of 800-number calls SSA receives, even  relatively small percentages of callers encountering agent discourtesy  could result in tens of thousands of callers not getting the service they  deserve. Thus, we believe that SSA can benefit from having uniform  procedures for documenting and assessing customer complaints and have  added additional information for further clarification. Experts believe that  paying attention to customer complaints, however minor, and working to  quickly resolve them is important to building relationships with  customers. In addition, having information on complaints helps identify  recurring problems and potential fixes as well as help prevent future  occurrences. Under SSA’s current practices, because the decision to  document a complaint lies with the individual agent handling the call,  customers contacting the 800 number have no assurance that SSA will  review the merits of their complaints. Routinely documenting and  assessing customer-initiated feedback could help the agency identify areas  of concern to callers and reinforce the agency’s commitment to providing  quality “citizen centered” service.
While we understand SSA’s concerns about resource constraints, we  maintain that SSA can implement a system to document complaints using  existing mechanisms, such as its 800-number feedback form and Internet  form for complaints reported to its 800 number and Web site, respectively.  As we state in the report, SSA already devotes time and staff to the  documentation and handling of customer-reported complaints; however,  such efforts are not done routinely. SSA states that its agents provide more  efficient service when they keep the caller on the phone until the caller’s  business and all agent actions are completed.  We believe routinely  documenting callers’ concerns would take no more time than completing  callers’ other business. Further, the information could be collected  uniformly in an electronic format that would facilitate analysis that could  be used to improve service. As others have pointed out, a good system for  managing complaints should be comprehensive, yet simple. Finally, we  believe that understanding and responding to customer complaints are  integral to the delivery of quality customer service.
SSA’s comments are reproduced in appendix II. SSA also provided  technical comments, which we have incorporated in the report as  appropriate.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies of this report to the interested  congressional committees and the Commissioner of SSA and will make  copies available to others upon request. In addition, the report will be  available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (201) 512-7215 or bovbjergb@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this study were to evaluate SSA’s actions for ensuring  callers have ready access to 800-number services and receive accurate and  courteous service from agents. To do this, we reviewed published works  that included the National Performance Review benchmarking reports to  identify industry benchmarks in areas key to our work and issues  surrounding call center services. We also reviewed GAO and the Office of  the Inspector General (OIG) reports and SSA annual performance plans to  identify what is currently known about SSA telephone service operations.  To evaluate the quality of the 800-number service, we compared telephone  system performance data compiled by a contractor for SSA and SSA’s  Office of Quality Assurance and Performance Assessment (OQA)  assessments of agent accuracy and courtesy to SSA’s established  standards and, where applicable, to industry benchmarks. We used  performance data from OQA’s monitoring of agents for fiscal years   1998 through 2003 and from OQA’s 800-number customer satisfaction  surveys conducted fiscal years 2001 through 2003. We reviewed OQA’s  management reports for these activities.
To develop information on the actions SSA takes at the headquarters level  to ensure quality 800-number telephone service, we reviewed documents  related to (1) SSA’s forecasts of call volumes and projected staffing levels  for auxiliary agents; (2) services offered using the automated menu;   (3) vendor-contracted services for the 800-number telephone systems  hardware, software, and performance data; and (4) requirements for  training agents, monitoring agent performance, and agent courtesy to  callers. We interviewed officials in the Office of Telephone Services to  obtain an understanding of the general operation of the 800-number  telephone system including the routing of calls; the compilation of  performance data; and SSA’s actions to monitor the performance of the  800-number system and of the vendor. We also interviewed OQA officials  to obtain more detailed information on procedures for monitoring agents  and surveying 800-number callers. In addition, we reviewed some  complaints reported by the public over the agency’s Web site and  interviewed officials in the SSA’s Center for Program Support to discuss  practices for handling complaints.
We visited six call centers to observe the 800-number service operations at  the regional and call center levels. At the locations visited, we observed  officials monitoring their centers’ call traffic and agent availability in real  time, officials monitoring agents handling live calls, and agents handling  live calls from customers. We reviewed documentation call center officials  provided on agent training, monitoring of agents, agent-related complaints  received, and disciplinary actions taken against agents. We interviewed  regional and call center officials having line-management, supervisory,  floor support, monitoring, and call-handling responsibilities to obtain  information on call center operations and their experiences in providing  telephone services and serving the public. The call centers we selected  varied in the frequency and volume of calls they handled—three handled  calls routinely and three on a back-up basis—and are not representative of  call centers SSA-wide.
To assess caller access and the reliability of the 800 number, we  interviewed SSA officials and contacted selected vendor staff to obtain  documents and data on the 800-number management and operations. SSA  uses the management data and information supplied by the vendor to  track all calls and transactions on the network, including data on overall  incoming calls and information on automation and to determine whether  calls were routed to an SSA call center or to a busy message. The vendor’s  reporting system has internal alarms running on each server and an  application that periodically checks each server’s vital functions, capacity,  and environmental operating conditions against a predetermined set of  normal operational conditions. Upon receipt, SSA runs the vendor- supplied data through a multistep automated procedure that backs up the  data, creates data storage files, extracts data to be stored in other datasets,  and recreates the data in a readable format. As part of this process, SSA  checks each record to ensure that all area codes are good, all phone  numbers are properly formatted, and all answering telephones originate in  the 800-number network. The vendor also generates separate reports on  automated services and agents. SSA reviews the reports and compares the  results with historical trends. Although SSA has no additional means of  verifying the reliability of the vendor-provided data or the results that  appear in report field outputs, both SSA and the vendor maintain that  these data are accurate and the vendor states that SSA has the source data  it needs to assess network performance. We reviewed SSA performance  data related to access and determined the data to be sufficiently reliable  for the purposes of this report.
To assess the reliability of OQA’s monitoring assessments of agents’  performance, we examined data reliability issues identified in an OIG  report and interviewed OQA officials knowledgeable about the monitoring  process and resulting data. In addition, we reviewed documentation and  training materials, including monitoring instructions, evaluation data entry  forms and desk aids, corrective action and evaluation feedback forms, and  information regarding the statistical sampling of calls. In evaluating OQA’s  sampling and weighting methodology, we determined that OQA’s  methodology for monitoring agents’ payment and service accuracy  appears to adequately represent the population of telephone calls.   Approximate confidence intervals were produced by OQA using standard  formulas for proportions based on a simple random sample.  As OIG  previously reported,  we also found that decisions regarding payment  accuracy and service accuracy continue to be unverifiable because SSA  does not maintain documentation of all monitored calls. We determined  that the data were sufficiently reliable for our purposes, given these  limitations.
To assess the reliability of the survey of 800-number callers, we  interviewed OQA officials about the survey and resulting data and  reviewed documentation on the survey methodology, sampling, response  rates, and sampling variability. We also reviewed a report contracted by  the OIG regarding this measurement of customer satisfaction. This report  concluded that the 800-number caller survey produced a reliable  measurement of callers’ views of agent courtesy for the period measured,  but that because the survey was administered only twice a year, it was  unlikely that the survey results matched the true customer satisfaction  across the entire year. Because the survey was recently limited to being  conducted during a single 4-week period in March, we found that the  survey results continue to be unrepresentative of callers’ responses  throughout the year. We believe that seasonal events could affect  customer satisfaction in different ways throughout the year. The survey  response rate during the period 2001 through 2004, ranged from 53 percent  to 71 percent. Although response rates within these ranges are not  unexpected for this kind of telephone survey, it should be noted that as  the response rate decreases, the certainty that the survey results represent  the universe decreases. We determined that the survey data are  sufficiently reliable for providing a general indication of customer  satisfaction, for the specified periods of administration.
We conducted our work at SSA headquarters, Baltimore, Maryland; at  regional offices in Birmingham, Alabama; Kansas City, Missouri; and  Richmond, California; and at two call centers in each region. We  conducted our work from September 2004 through July 2005 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Social Security Administration
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: The following individuals made important contributions to this report:  Shelia Drake, Assistant Director, Jacquelyn Stewart, Analyst-in-Charge,  Susan Bernstein, Michelle Fejfar, Jonathan McMurray, and Roger Thomas.