Immigration Services: Better Contracting Practices Needed at Call Centers
Released: 2005-06-30T00:00:00

Letter

Background

The Homeland Security Act of 2002 established USCIS within DHS. USCIS is responsible for several functions transferred on March 1, 2003, from the former Immigration Services Division of the Immigration and Naturalization Service (INS) under the Department of Justice. These functions include providing services or benefits to facilitate entry, residence, employment, and naturalization of legal immigrants; processing applications for U.S. citizenship/naturalization; and rendering decisions about immigration-related matters.

Call Center Operations

The USCIS Information & Customer Service Division is responsible for operating the National Customer Service Center (NCSC), which was established in 1997 to provide nationwide assistance by telephone to customers calling about immigration services and benefits. When a customer calls the NCSC toll-free number (1-800-375-5283), the call is received by the interactive voice response system. The system features automated, self-service options 24-hours a day, 7 days a week. If the system cannot address a customer’s concerns or needs or if a customer requests live assistance, then the call is generally routed to one of the four NCSC contract call centers, known as Tier 1. These four centers are operated by the contractor, Pearson. If a question posed by a customer is particularly complex or otherwise cannot be answered at the Tier 1 level, the call is transferred to one of the two USCIS-operated call centers, known as Tier 2. Figure 1 shows the organization of NCSC, including the call centers.

In fiscal year 2004, almost half of 21.1 million calls made to NCSC were handled and completed by the interactive voice response system and the rest were generally routed to Tier 1. Customer service representatives (CSR) at Tier 1 respond to inquiries in English or Spanish. The CSRs focus primarily on providing administrative information to customers by using a series of scripts provided by USCIS. For example, if a customer needs what USCIS considers basic information, such as USCIS local offices’ hours of operations, eligibility requirements, and procedures to follow, such questions are to be answered by CSRs at Tier 1 call centers using specific scripts. In addition, CSRs are to refer customers to USCIS service centers and local offices, for such things as changes of address and appointment scheduling at USCIS application support centers. (Some of these tasks may alternatively be performed by customers through the USCIS Web site—www.uscis.gov.) As of April 2005, the four Tier 1 call centers employed over 450 CSRs. Figure 2 shows CSRs processing calls at a Tier 1 call center.

At the two USCIS-operated Tier 2 call centers, calls are handled by immigration information officers (IIO)—immigration specialists with in- depth knowledge of immigration laws, non-immigrant visas, naturalization, asylum and refugee status, and other related policies and procedures. As of April 2005, the Tier 2 call centers operated by USCIS had 111 IIOs. About 5 percent, or about 590,000, of the calls going to Tier 1 CSRs were rerouted to Tier 2 IIOs in fiscal year 2004. Figure 3 shows the call volume handled by the interactive voice response system, Tier 1 call centers, and Tier 2 call centers during fiscal year 2004.

Performance-Based Service Contract

In January 2002, USCIS awarded a performance-based service contract for the management of four Tier 1 call centers. In making this award, USCIS obtained acquisition services from the Department of Veterans Affairs (VA), and the contracting officer who signed and was responsible for administering the contract was a VA employee working on behalf of USCIS. The contracting officer’s technical representative (COTR), a USCIS employee, was also responsible for administering the contract. In commenting on a draft of this report, DHS noted that by agreement of both the VA and USCIS, on April 20, 2005, USCIS assumed responsibility for administering the contract. The contract was awarded for a base year, beginning on June 1, 2002, plus 4 option years (1-year renewable extensions of the contract, three of which have been exercised as of June 2005). Through calendar year 2004, USCIS paid $64.6 million to the contractor for the Tier 1 call center operations.

According to the Office of Federal Procurement Policy (OFPP) in the Office of Management and Budget, performance-based service contracts are designed to focus on results. Their purpose is to ensure that contractors are given the freedom to determine how to meet the government’s performance objectives, that appropriate quality levels are achieved, and that payment is made only for services that meet these levels. This type of contract is to emphasize standards for customer service and measurement of performance and may offer financial incentives, both positive and negative, to encourage quality performance. According to OFPP, call centers are suited to this type of contract because, among other things, they emphasize achieving results by meeting customer service standards. According to OFPP, with performance-based service contracts, incentive payments made to an independent contractor are to be contingent on the contractor’s ability to meet the government’s performance standards; the contract does not specify how those standards are to be met. Thus, the contractor retains discretion in determining how to meet performance standards specified in the contract, for example, how many CSRs to hire to ensure calls are answered within a contractually specified time. Other elements suggested for using a performance-based service contract include (1) identifying the agency’s needs and addressing those needs with performance requirements that describe required service results; (2) establishing performance standards that describe the required performance level; and (3) establishing a quality assurance plan for assessing contractor performance in order to ensure that the contractor has performed in accordance with the standards.

USCIS Established Performance Measures to Assess Contractor’s Call Center Performance

USCIS used a multi-faceted approach to monitor and evaluate the quality of information and service provided by CSRs to customers calling contractor-operated Tier 1 call centers. This approach used seven performance measures. USCIS obtained performance data from the contractor’s monitoring of selected telephone calls; customer satisfaction surveys; and a telecommunications vendor (telephone company). In addition, USCIS used an independent consulting firm to monitor CSRs’ telephone calls and conduct a “mystery shopper” program assessing CSRs’ responses to customers.

USCIS Set Call Center Performance Measures for the Contractor to Meet

In order to monitor and evaluate the performance of the four contractor- operated Tier 1 call centers, USCIS planned to use seven performance measures. These measures were to evaluate the quality of customers’ telephone interactions with CSRs; the accuracy of information provided to callers over the telephone; the accuracy of callers’ information recorded by CSRs; callers’ levels of satisfaction; how quickly CSRs handled calls (two measures); and the number of calls abandoned by customers put on hold. According to USCIS officials, USCIS established the performance measures based on a review of industry standards for both government and private-sector call center operations. The measures were described in a section of the contract called the Performance Requirements Summary (PRS).

Under the PRS, these performance measures comprised one of three components upon which the contractor’s performance score was based. The other two components were the standard, or goal, set for each measure, identifying the performance levels the contractor was expected to meet (e.g., callers will wait an average of 30 to 36 seconds before their calls are answered), and the performance calculation that USCIS would use to analyze performance data (e.g., total delay of all calls divided by the total number of calls). The PRS listing of the seven performance measures included a “sample calculation” for each of the measures, and stated that “actual calculations to be determined during Contract negotiations.” USCIS officials said they intended to negotiate and finalize the calculations after a 4- to 6-month phase-in period, and the contract was awarded with this provisional language.

As to the performance measures and their related standards or goals, three of USCIS’s performance measures are call quality monitoring, accuracy of information provided, and accuracy of capturing information. Data on these measures are to be collected by the contractor’s quality assurance staff, who are to randomly monitor two calls per day for each CSR. (CSRs are not to know when they are being monitored.) The data collected are to be reported to USCIS on a monthly basis. Details on these three measures follow: Call quality monitoring. Calls are to be monitored by the contractor’s quality assurance staff to assess the CSRs’ “soft skills,” that is, their ability to interact with customers, establish customer rapport, maintain composure during a call, speak with clarity and professionalism, and other factors. Call quality monitoring data are to be captured on a standardized form. CSR responses for each of nine different “soft skills” are scored as percentages, with scores for the most highly valued skills, such as “active listening”—that is, whether the CSR was deemed to be attentive when listening to the customers—given more weight than the scores for other skills. The nine scores (i.e., percentages) are combined for a total “soft skills” score, with 100 percent as the highest possible score. The performance standard stated in the PRS for this measure is that all calls monitored achieve an average score of 90 percent to 95 percent after the nine “soft skills” scores (i.e., percentages) for each call are combined. (See app. II for additional details on the criteria and methodology used to determine soft skills scores.)

Accuracy of information provided. Calls are to be monitored by the contractor’s quality assurance staff to determine, among other things, whether CSRs provided accurate and complete responses. Using a standardized form, the staff score CSRs on five different efforts, such as whether the CSR used software tools appropriately and whether, when the callers were asked directly, they indicated that their needs had been satisfied. The five efforts are scored as percentages, with more weight given to the scores for certain efforts, such as “provides complete response.” The scores are then combined for a total “accuracy of information provided” score, with 100 percent as the highest possible score. The performance standard stated in the PRS for this measure is that all calls monitored achieve an average score of 95 percent to 97 percent after the five accuracy scores (i.e., percentages) for each call are combined. (See app. II for additional details on the criteria and methodology used to determine accuracy of information provided.)

Accuracy of capturing information. Calls are to be monitored by the contractor’s quality assurance staff to determine, among other things, whether CSRs accurately record and verify the callers’ information. The staff assess this measure by scoring four efforts, including whether a referral to a local USCIS service center or local office was completed appropriately and correctly. The four efforts are scored as percentages, with more weight given to the scores for certain efforts, such as “verifies caller’s information.” The scores are then combined for a total “accuracy of information provided” score, with 100 percent as the highest possible score. The performance standard stated in the PRS for this measure is that all calls monitored achieve an average score of 95 percent to 97 percent after the four accuracy scores (i.e., percentages) for each call are combined. (See app. II for additional details on the criteria and methodology used to determine accuracy of capturing information.)

A fourth performance measure of call quality—customer satisfaction— was assessed by an independent consulting firm. Customer satisfaction surveys were conducted on a monthly basis to determine if customers were satisfied with the service that CSRs provided. At least 375 callers are to be randomly selected to be interviewed each month from a population of 10,000 randomly identified callers who called within the 30 days prior to the survey. To measure satisfaction with CSRs, customer responses to four interview questions about CSRs are compiled, and the overall percentage of respondents indicating satisfaction is calculated. The performance standard stated in the PRS for this measure is 80 percent to 85 percent of the customers surveyed indicating overall satisfaction with the CSRs’ service. (See app. III for additional details on the criteria and methodology used to determine customer satisfaction.)

Three other performance measures involve the collection of statistical data by the telecommunications vendor for determining how quickly calls are answered. The performance measures and standards in the contract for assessing how quickly CSRs answered customers’ calls are as follows: Service level. The telecommunications vendor under contract with USCIS is to collect information on the number of calls answered by CSRs in 20 seconds or less, that is, the number of callers who spoke to a CSR within 20 seconds after getting through the interactive voice response system. The performance standard stated in the PRS for this measure involves two factors: half-hour increments and the length of time it took CSRs to answer calls. The standard is that for 80 percent to 85 percent of the half-hour increments measured, 80 percent of the calls are to be answered in 20 seconds or less.

Average speed of answer. The telecommunications vendor under contract with USCIS is to collect information on the length of time it takes for CSRs to answer customers’ calls after they are routed to Tier 1 by the interactive voice response system; that is, how long callers are on hold before a CSR answers their call. The performance standard stated in the PRS for this measure is that, for all calls routed to Tier 1, callers will wait an average of 30 seconds to 36 seconds.

Abandoned calls. The telecommunications vendor under contract with USCIS is to collect information on the number of calls abandoned by customers after getting through the interactive voice response system and waiting for a CSR to answer, that is, the number of times that customers hang up the telephone while waiting for a CSR. The performance standard stated in the PRS for this measure involves two factors: half-hour increments and how frequently callers abandon their calls. The standard is that for 85 percent to 95 percent of the half-hour increments measured, 1 percent to 2 percent of the calls are expected to be abandoned before a CSR answers.

The contract stated that the contractor would be eligible to earn financial incentive awards if the average monthly performance met or exceeded the standards on a quarterly basis at each call center, and allowed USCIS to make deductions from payments to the contractor if the average monthly performance fell below the standards. According to the contract, the contractor is not eligible for an incentive award for a particular quarter if one of the performance standards is not met by one call center, and USCIS may make a deduction from payments to the contractor in that case. In addition, USCIS may, at its sole option, elect to include or waive financial incentives as it deems appropriate.

USCIS Carried Out Additional Quality Assurance Efforts

In addition to the performance data collected by the contractor’s own quality assurance staff, an independent consulting firm, and the telecommunications vendor, USCIS took two additional steps to measure call center performance for quality assurance purposes. First, to help ensure that the contractor’s scoring of call-quality performance measures was reliable, USCIS used another independent consulting firm to validate the results of the contractor’s efforts by monitoring two calls per month for each CSR. Data were gathered and provided to USCIS on a monthly basis. (See app. IV for additional details on the criteria and methodology used by the independent consulting firm to conduct call monitoring.)

Second, in April 2003, USCIS engaged the same independent consulting firm to carry out a “mystery shopper” program to assess the completeness and accuracy of CSRs’ answers to callers. Under this program, an independent consultant places random calls—1,200 each month—to Tier 1 call centers using various scripts provided by USCIS. As of April 2005, the scripts used in these calls covered 32 different scenarios, or types of calls, and 100 new scenarios were being developed. The calls are conducted in English and Spanish. (See app. V for an example of a mystery shopper scenario.)

USCIS Evaluated Contractor’s Performance but Suspended Use of Financial Incentives for More Than 2 Years Due to Performance Measurement Dispute

USCIS did not reach agreement with the contractor on how to apply the performance measurement requirements described in the PRS before awarding the performance-based service contract. USCIS suspended all financial incentives, positive or negative, while the parties negotiated this issue over a period of about 16 months without reaching agreement. After negotiations were abandoned, USCIS determined that, for the fourth quarter of 2004, the contractor had failed to meet four of seven performance measures and merited a payment deduction. The contractor disagreed on the grounds that the performance measurements had not been finalized and that changes in call center workloads affected the basis for applying financial incentives. In a separate matter, USCIS failed to ensure that all contractual, regulatory, and GAO standards pertaining to the documentation of the contractor’s performance were fulfilled.

USCIS Awarded the Call Center Contract without Finalizing Performance Measurement Requirements

The performance measurement requirements described in the PRS were not completely finalized before the contract was awarded. The language referring to “sample” calculations for determining how performance would be measured remained in the contract after it was in force. In commenting on a draft of this report, DHS said that at the time of the contract award, USCIS management believed it was appropriate to let the winning vendor have some input into the performance measurement methodology since this contract represented a transition to performance-based contracting for call center operations.

The negotiations between USCIS and the contractor on this issue began in January 2003 (after a phase-in period) and continued intermittently until April 2004, when they were abandoned. While negotiations were taking place and after they were abandoned, USCIS obtained monthly data relating to the contractor’s performance on the seven performance measures and compared those data to the standards. USCIS considered the measures and standards themselves to be nonnegotiable; the contractor, on the other hand, considered them as part of the “sample calculations” and, thus, negotiable. For over 2 years, USCIS did not use any of the resulting performance scores for the purpose of calculating financial incentive awards or payment deductions under the contract because the terms of the PRS remained unresolved between the parties.

The contractor maintained that the performance scores were “potential scores” and were to be used by the parties in reaching an agreement on how to structure the PRS.

On September 1, 2004, the contracting officer, representing USCIS, sent a letter to the contractor advising that USCIS would begin evaluating the contractor’s performance and determining a financial incentive award or payment deduction for the fourth quarter of the calendar year (October 1 through December 31). USCIS officials told us they decided to take this action because they had concluded that negotiations with the contractor were unlikely to result in an agreement on the PRS. The contractor objected to USCIS’s decision to carry out this evaluation. By letter dated November 29, 2004, the contractor stated that, under the terms of the contract, USCIS could not unilaterally determine the performance measurement requirements because all aspects of the requirements were negotiable, including the performance standards.

The contractor further stated that an evaluation of its performance must take into account certain changes that took place to the work required under the contract. For example, the contractor stated that the number of USCIS-provided scripts, containing information for CSRs to address callers’ inquiries, had grown to more than 2,300 pages from approximately 400 script pages in June 2002. According to the contractor, these changes significantly increased the average amount of time needed to handle a call and affected the contractor’s ability to meet the performance standards imposed by USCIS. According to the contractor, USCIS’s unilateral imposition of performance measurement requirements that did not account for the changed work requirements was inconsistent with Federal Acquisition Regulation (FAR) 16.402(g), which provides that “t is essential that the Government and contractor agree explicitly on the effect that contract changes (e.g., pursuant to the Changes clause) will have on performance incentives.”

Nevertheless, by letter dated February 11, 2005, USCIS’s contracting officer notified the contractor of the evaluation results for the period of October through December 2004. The results showed that the contractor met the standards for three of the seven performance measures and did not meet the standards for the other four measures. USCIS determined that, as a result of this performance, payments due to the contractor for services would be reduced. The letter noted that the contractor could submit its own data regarding performance during this period. Following the review of any data submitted, USCIS would take action to make the appropriate payment deduction, waive the payment deduction, or pay an appropriate incentive award.

The contractor requested, by letter dated February 25, 2005, that USCIS waive implementation of the financial incentives, both positive and negative. The contractor reiterated its position that USCIS’s unilateral implementation of the performance measurement requirements as currently written in the contract, without sufficient regard for substantial changes to the contract and the changing nature of the program, was not appropriate. The contractor stated that it was ready to resume negotiations on this subject so that fair and equitable financial incentives would be established. The contractor further stated that it had determined the payment deduction was incorrectly calculated by USCIS.

USCIS’s contracting officer responded, by letter dated April 15, 2005, that the government would not agree to waive implementation of the financial incentives and a deduction would be made from the next payment to the contractor. The letter stated that USCIS did not unilaterally create and impose the performance measurement requirements, which were included in the negotiated contract that USCIS and the contractor agreed to. Regarding the contractor’s assertion that the average amount of time needed to handle calls had significantly increased, the letter noted that the performance measurement requirements would apply regardless of the average length of calls at any given time.

According to FAR and OFPP guidance on performance-based service contracting, the precise method for measuring performance should have been agreed upon between USCIS and the contractor before the contract was signed and implemented. FAR § 16.401 states that performance-based service contracts should establish “reasonable and attainable targets that are clearly communicated to the contractor.” According to OFPP, performance measurement techniques (i.e., how performance will be assessed to determine whether standards have been met) are essential elements of performance-based service contracting and should be clearly stated. In addition, according to OFPP, performance- based service contracts emphasize that all aspects of an acquisition be structured around the purpose of the work to be performed, that appropriate performance quality levels are achieved, that payment is made only for services that meet these levels, and that financial incentives are awarded to encourage quality performance.

USCIS Plans to Solicit New Tier 1 Call Center Contracts with Changes to Improve Performance

Although the disagreement between the two parties had not been resolved, USCIS exercised its option to extend the current call center contract for another year through May 31, 2006, to allow time to solicit and award new call center contracts. The exercise of this option has no effect on the contract’s performance measurement terms, which is the source of the parties’ dispute. USCIS officials said they plan to award new performance- based service contracts for Tier 1 operations to two vendors, with the two vendors fully operational by June 2006, to improve the handling of customers’ calls to Tier 1.

USCIS officials told us they intend for the new contracts to include certain changes meant to improve Tier 1 call center operations and to incorporate OFPP guidance on performance-based contracting. USCIS officials told us that, unlike the current contract, the new PRS will clearly specify how contractor performance will be assessed and will not leave any terms open for post-award negotiation. In addition, USCIS officials said the new contracts will include independent call monitoring and the mystery shopper program as performance measurement tools to assess the quality of the Tier 1 CSRs’ responses to customers, including the accuracy and reliability of the information provided. At the time of our review, USCIS officials said that the solicitation was going through DHS’s contract review process and DHS had not issued the solicitation for a new contract containing these changes. DHS said in its comments on a draft of this report that the solicitation was with the DHS Procurement Office for review and issuance.

USCIS and Contractor Did Not Fully Document Quality Assurance Activities

As part of its quality assurance responsibilities under the current contract, USCIS is to keep written records of observations about the contractor’s performance based on periodic evaluations comparing performance data to standards in the PRS. USCIS’s contracting officer’s technical representative (COTR), who is responsible for administering the contract, is to use these written observations to notify the contractor if there are deficiencies—specifically, if the contractor does not meet the performance standards. The contractor is required to sign and date such observations to acknowledge that the COTR apprised it of any deficiencies.

USCIS and contractor officials said they met at least quarterly (monthly, since October 2004) to discuss performance, performance data, and other items. USCIS officials said they provided the contractor with documentation containing performance and other data to discuss at these meetings. USCIS officials said some of this documentation identified performance deficiencies. However, contractor officials said they viewed the performance data as “potential scores” to be considered during negotiations. To the extent that USCIS considered the performance data as notification of deficiencies, it did not follow contractual procedures requiring the COTR to obtain the contractor’s signature acknowledging notification of the deficiencies. In addition, neither USCIS nor the contractor kept minutes of these meetings.

According to FAR § 46.104(c), the government should maintain, as part of the performance records of a contract, suitable records reflecting the nature of its contract quality assurance actions. With respect to any performance deficiencies, the government’s records should include, among other things, the number and type of defects observed and any actions to correct deficiencies. Further, according to GAO’s standards for internal control in the federal government, for an agency to run and control its operations, it must have relevant, reliable information relating to internal events. All transactions and other significant events need to be clearly documented, and the documentation should be readily available for examination. This information should be recorded and communicated to management and others within the agency who need it to carry out their responsibilities. GAO’s standards provide a framework for establishing and maintaining internal control and for identifying and addressing major performance challenges. Appropriate and effective internal control is a key factor in helping agencies better achieve program results.

The contract also requires the contractor to provide a quality assurance plan. The plan that was developed by the contractor describes the contractor’s approach and strategy for ensuring the delivery of high-quality service. As part of the plan, the contractor is to conduct formal, biweekly internal performance review meetings to help with the identification and correction of performance deficiencies. These meetings are to be attended by contractor and USCIS officials, with contractor staff reporting on quality performance issues, and are to be in addition to the quarterly (now monthly) meetings discussed above. Minutes of the meetings are to identify action items, responsibilities, and solution time frames, and the minutes are to be published for USCIS review. However, a contractor official said that these meetings never took place. According to both contractor and USCIS officials, the quarterly meetings were used to discuss operations and performance and to focus senior management attention on any performance issues.

Under FAR § 37.602-2, the government’s quality assurance surveillance plans should include actions to help ensure that the contractor carries out its quality control obligations. By failing to ensure that the contractor held and documented performance review meetings as required by the contractor’s quality assurance plan, USCIS did not meet its quality assurance obligations under FAR § 37.602-2 and GAO’s internal control standards. In addition, USCIS’s failure to obtain the contractor’s written acknowledgment of USCIS-identified performance deficiencies did not meet the notification procedures established by the contract, documentation requirements of FAR § 46.104(c), and GAO’s standards for internal control.

In its comments on a draft of this report, DHS noted that the contract was administered by a component of the VA until April 20, 2005, and DHS said the VA was provided with documentation discussed at quarterly and monthly performance assessment meetings between USCIS and the contractor. According to DHS, the lack of a clear understanding between USCIS and the VA regarding their roles contributed to the fact that formal documentation and evaluations were not always properly maintained and formally transmitted to the contractor. DHS acknowledged that the agency procuring a service is ultimately responsible for the contract and, thus, USCIS should have clarified its and the VA’s roles.

USCIS Has Used Results of Monitoring Efforts to Identify Opportunities to Improve Customer Service and Call Flow at All Call Centers

USCIS used contractor performance data, including the results of surveys, call monitoring, and the mystery shopper program, to identify opportunities to improve customer service, including improving call- response times, help CSRs and IIOs better respond to customer inquiries, and manage the flow of calls into call centers. Following are examples of initiatives that USCIS recently implemented, or was planning to implement, as of April 2005. It is too early to assess the impact of these initiatives.

USCIS implemented “intelligent call routing” for Tier 1. With “intelligent call routing” in place since February 2004, telephone calls routed from the interactive voice response system to Tier 1 are now routed to the next available CSR, at any of the four Tier 1 contractor- operated call centers. Previously, telephone calls to Tier 1 were routed to the next available CSR in the call center that resided in the same geographic region of the country as the caller. USCIS officials said that by June 2005 they will also have implemented “intelligent call routing” for calls transferred from Tier 1 to the next available IIO at either of the two Tier 2 call centers.

USCIS implemented “overflow routing.” USCIS started its “overflow routing” initiative in October 2004, enabling certain general call types, identified as “English Other” and “Spanish Other,” to be routed directly from the interactive voice response system to Tier 2 USCIS-operated call centers, bypassing the Tier 1 contractor-operated call centers. Previously, all calls handled by IIOs at Tier 2 were first routed from the interactive voice response system to Tier 1, where CSRs then transferred the calls to Tier 2. USCIS officials said they expect the change will result in 1 to 5 percent of all calls being routed directly to Tier 2, which should help when Tier 1 CSRs cannot handle the call volume.

USCIS implemented interactive voice response system routing of certain telephone calls to USCIS service centers. USCIS changed its automated interactive voice response system in December 2004 so that certain types of customers’ telephone calls—for example, certain issues concerning new permanent residents, cases already approved or denied, and pending cases—are now routed directly to USCIS service centers, bypassing CSRs at Tier 1. Previously, all customers’ telephone calls that needed to be handled by USCIS service centers were routed by the interactive voice response system to Tier 1. Then, after talking with the customers, the CSRs referred the customers to the service centers (whose employees have access to case paperwork) via e-mail. CSRs were allowed to transfer customers’ telephone calls to service center personnel only when customers requested emergency and expedited handling of applications.

USCIS implemented a portfolio management system. Private attorneys, paralegals, and other representatives can use the USCIS Internet Web site to check the status of their clients’ immigration cases using a USCIS receipt number. Under the system, USCIS also notifies the representatives via e-mail when a case status changes; for example, when actions are taken, such as the approval or denial of an application. As of April 2005, over 300,000 customers, attorneys, and other representatives had used this system.

USCIS said it is planning to implement a referral management system.

Currently, Tier 1 CSRs send, via e-mail, service request referrals to USCIS service centers and local offices for customers who call wanting to change addresses, schedule and reschedule appointments at application support centers, order forms, and resolve problems. After a referral is made, NCSC does not know whether the service center or local office responded to the customer in a timely manner or even responded at all. To better monitor this process, USCIS plans to implement a referral management system, with such service request referrals placed in a database and assigned a tracking number. The system is to (1) determine the proper service center or local office to process the referral, (2) assign the case to an adjudicator, (3) update the case on a daily basis, and (4) report once a month on case status. The referral management system is planned to be accessible to customers on USCIS’s Internet Web site so they can make and track their own service request referrals. In addition, customers without Internet access are to be able to call on the telephone and CSR’s will access the USCIS Web site and create referrals for them. USCIS plans that the referral management system will be fully operational during the summer of 2005.

USCIS is planning a customer service portal on USCIS’s Web site. USCIS has a long-term goal of giving customers Internet access to information contained in the “scripts” used by Tier 1 CSRs to answer customers’ questions. USCIS plans to establish a customer service portal on the USCIS Internet Web site, providing access to the information. The goal is to let customers with Internet access look up information themselves without having to call NCSC on the telephone, navigate the interactive voice response system, and wait for CSR’s to answer. USCIS had not set a time frame for implementing this initiative.

Conclusions

Immigration call centers are a vital information referral source used millions of times by immigrants and other interested parties seeking to obtain needed documents, regulatory information, up-to-date status information on immigration-related benefits and applications, and other information. To ensure that it serves its customers effectively and efficiently, USCIS appropriately used a performance-based contract, but its failure to finalize all aspects of the performance requirements before the contract was awarded hampered its ability to exercise performance incentives in the contract. As a result, USCIS lost the opportunity during the life of the contract to help ensure that it received the maximum level of service from the contractor.

In addition, USCIS did not meet standards promulgated by federal acquisition regulations, GAO, and the contract itself pertaining to documenting the contractor’s performance between 2002 and 2004, and adequately documenting notification of the contractor when the government perceived deficiencies in its performance. Failure to generate adequate documentation could impair USCIS’s ability to conduct future contract negotiations and to preserve a complete and reliable record of contract performance needed to ensure accountability.

Recommendation for Executive Action

To improve USCIS’s efforts for evaluating contractor performance and encourage quality services at call centers, we recommend that the Secretary of Homeland Security require the Director of USCIS take the following two actions: (1) finalize contract terms related to specific performance measurement requirements before awarding new performance-based call center contracts; and (2) maintain readily available written records of performance assessments and performance evaluation meetings with the contractor.

Agency Comments and Our Evaluation

DHS and the contractor provided formal comments and technical comments on a draft of this report, which we have incorporated, as appropriate. In its formal comments, DHS generally agreed with our recommendations. DHS said the draft solicitation for the new contracts specifically identifies performance requirements that are non-negotiable. DHS further stated that, as recommended by GAO, written records of performance assessments and performance evaluation meetings will be maintained and readily available for review by all interested parties. In its formal comments, the contractor provided additional language to further clarify this report. The contractor said the report accurately summarizes the complex nature of CIS’s call center program and several challenges created by significant post-award changes to that program. DHS’s and the contractor’s formal comments are shown in appendixes VI and VII, respectively.

We are sending copies to the Director of USCIS and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions regarding this report, please contact me at 202-512-8777 or jonesp@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VIII.

Appendix I: Scope and Methodology

To determine what performance measures the U.S. Citizenship and Immigration Services (USCIS) established to monitor and evaluate the performance of contractor-operated call centers, we interviewed USCIS headquarters officials in Washington, D.C.; Tier 1 contractor officials in Arlington, Virginia, and at a contractor-operated call center; and an official representing an independent consulting firm under contract to USCIS and located in Fairfax, Virginia. We also collected and analyzed pertinent USCIS and contractor documentation. We collected and analyzed information on the various types of monitoring and evaluation programs used by USCIS, including internal call monitoring, independent call monitoring, customer satisfaction surveys, the mystery shopper program, and telephone call data provided by a telecommunications vendor.

To find out how USCIS used the performance measures to evaluate the contractor’s performance, we interviewed USCIS headquarters officials in Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia. We also collected and analyzed pertinent USCIS and contractor documentation.

To determine what actions, if any, USCIS took or planned to take to strengthen call center operations, we interviewed USCIS headquarters officials in Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia, and at a contractor-operated call center. We also collected and analyzed pertinent USCIS and contractor documentation.

We assessed the reliability of telephone call volume data provided to USCIS by a telecommunications vendor, as well as USCIS and contractor staffing data. To carry out our data reliability assessments, we (1) reviewed information about the data, systems that produced the data, and data quality control procedures, and (2) interviewed USCIS and contractor officials knowledgeable about the data as necessary. We determined that the call volume and staffing data were sufficiently reliable for the purposes of this report.

We conducted our work between May 2004 and May 2005 in accordance with generally accepted auditing standards.

Appendix II: Criteria and Methodology Used by Contractor to Measure Quality of Tier 1 Calls Monitored

Scoring range (0 – 3)

Section score (Percent)

Section score (Percent)

Scoring range (0 – 3)

Section score (Percent)

Appendix III: Criteria and Methodology Used to Measure Customer Satisfaction with Customer Service Representatives

USCIS and an independent consulting firm jointly developed a telephone survey to measure customer satisfaction with the three levels of NCSC call center service—interactive voice response system, Tier 1 CSRs, and Tier 2 IIOs. To carry out the survey each month, representatives of the independent consulting firm call 375 randomly selected customers. To assess the customers’ satisfaction with the CSRs, the representatives read several statements and ask questions for the customers to rate their experiences with CSRs. For the customer satisfaction performance measure required in the contract, USCIS collects and summarizes data on the customers’ responses to the four statements below. The customers are asked to rate their agreement with each of the statements using a scale of 1 to 7 (1 is strongly agree and 7 is strongly disagree). 1. The representative seemed to fully understand my questions. 2. The representative was polite. 3. The representative did not rush me. 4. The representative answered my questions promptly.

Appendix IV: Criteria and Methodology Used by Independent Consulting Firm to Measure Quality of Calls Monitored

An independent consulting firm scored CSRs on 23 separate quality assurance factors as follows.

Service orientation

Issue identification

Communications

Call management

Appendix V: Mystery Shopper Scenario Example

Appendix VI: Comments from the Department of Homeland Security

Appendix VII: Comments from Pearson Government Solutions

Appendix VIII: GAO Contact and Staff Acknowledgments

GAO Contacts

In addition to the above, Darryl W. Dutton, Ronald G. Viereck, Brian J. Lipman, Christine F. Davis, Amy L. Bernstein, and Michele C. Fejfar made key contributions to this report.