Immigration Services: Better Contracting Practices Needed at Call Centers
Published: 2005-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. Citizenship and Immigration Services (USCIS) bureau within the Department of Homeland Security (DHS) provides toll-free telephone assistance through call centers to immigrants, their attorneys, and others seeking information about U.S. immigration services and benefits. As the volume of calls increased--from about 13 million calls in fiscal year 2002 to about 21 million calls in fiscal year 2004--questions were raised about USCIS's ability to ensure the reliability and accuracy of the information provided at call centers run by an independent contractor. This report analyzes: (1) the performance measures established by USCIS to monitor and evaluate the performance of contractor-operated call centers; (2) how performance measures were used to evaluate the contractor's performance; and (3) any actions USCIS has taken, or plans to take, to strengthen call center operations.
section_title: What GAO Found
paragraphs: USCIS developed seven performance measures intended to assess the performance and overall quality of responses provided by customer service representatives at contractor-operated call centers. These measures include how quickly calls were answered and the accuracy of information provided. The contract between USCIS and its contractor stipulated that the contractor could earn financial incentive awards if the average monthly performance met or exceeded the standards on a quarterly basis at each of four call centers. Conversely, financial deductions could be made if the standards were not met. USCIS did not finalize the terms regarding how the contractor's actual performance would be calculated, or scored, before awarding the contract. This limited USCIS's ability to exercise performance incentives (positive or negative) because the parties could not reach agreement on performance terms. USCIS suspended the use of financial incentives while the parties negotiated the issue. Agreement was not reached after 16 months, however, USCIS determined that the contractor had failed to meet standards for 4 of the 7 performance measures in the fourth quarter of 2004 and took action to reduce its payments for services. The contractor objected, citing the lack of agreement on the performance measurements and the impact of workload increases, but USCIS disagreed and stated it would reduce payment. In a separate but related matter, USCIS failed to meet contractual, regulatory, and GAO standards pertaining to how the contractor's performance would be documented--especially with respect to any deficiencies. Finally, USCIS exercised its option to extend the call center contract through May 2006, to allow time to solicit and award new call center contracts. USCIS said it intends to finalize performance measurement terms in the new contracts. USCIS used contractor performance data it collected over the course of the contract to identify opportunities to improve customer service and call flow, among other things. Several initiatives were launched as a result.

section_title: Letter
paragraphs: The U.S. Citizenship and Immigration Services (USCIS) bureau within the  Department of Homeland Security (DHS) provides telephone assistance to  customers calling about U.S. immigration services and benefits. USCIS  customers—immigrants and their attorneys or family members,  employers, and others—can call toll-free for information on such topics as  how to become U.S. citizens, obtain work visas, initiate the naturalization  process, obtain legal forms, and determine the status of applications for  benefits. This telephone assistance service helps customers avoid making  visits to USCIS offices, which could require customers to take time off  from work and travel long distances. Telephone calls for assistance to  USCIS have increased in recent years, from about 13 million in fiscal year  2002 to about 21 million in fiscal year 2004. These calls are handled in a  variety of ways: by an interactive voice response system; by four call  centers managed by a private contractor (Pearson Government Solutions)  engaged by USCIS; and by two call centers operated directly by USCIS.
You expressed interest in how USCIS monitors and evaluates contractor- operated call centers to ensure that they operate reliably and provide  accurate information to their growing number of customers. This report  addresses the following: (1) What performance measures did USCIS  establish to monitor and evaluate the performance of contractor-operated  call centers? (2) How were these performance measures used to evaluate  the contractor’s performance? (3) What actions, if any, did USCIS take or  plan to take to strengthen call center operations?
To address these objectives, we obtained and analyzed information from  USCIS officials in Washington, D.C., from contractor officials in Arlington,  Virginia, and at one of four contractor-operated call centers. At these  locations, we interviewed officials and collected and analyzed pertinent  documentation, including descriptions of monitoring and evaluation  programs, contract requirements for call center operations, and  summaries of call center evaluation results. We assessed the reliability of  telephone call volume data provided to USCIS by a telecommunications  vendor, as well as USCIS and contractor staffing data. To carry out our  data reliability assessments, we (1) reviewed information about the data,  systems that produced the data, and data quality control procedures, and  (2) interviewed USCIS and contractor officials knowledgeable about the  data as necessary. We determined that these data were sufficiently reliable  for purposes of this report.
We conducted our work between May 2004 and May 2005 in accordance  with generally accepted auditing standards. (See app. I for details about  our scope and methodology.)
subsections: section_title: Background
paragraphs: The Homeland Security Act of 2002 established USCIS within DHS. USCIS  is responsible for several functions transferred on March 1, 2003, from the  former Immigration Services Division of the Immigration and  Naturalization Service (INS) under the Department of Justice. These  functions include providing services or benefits to facilitate entry,  residence, employment, and naturalization of legal immigrants; processing  applications for U.S. citizenship/naturalization; and rendering decisions  about immigration-related matters.
subsections: section_title: Call Center Operations
paragraphs: The USCIS Information & Customer Service Division is responsible for  operating the National Customer Service Center (NCSC), which was  established in 1997 to provide nationwide assistance by telephone to  customers calling about immigration services and benefits. When a  customer calls the NCSC toll-free number (1-800-375-5283), the call is  received by the interactive voice response system. The system features  automated, self-service options 24-hours a day, 7 days a week. If the  system cannot address a customer’s concerns or needs or if a customer  requests live assistance, then the call is generally routed to one of the four  NCSC contract call centers, known as Tier 1. These four centers are  operated by the contractor, Pearson. If a question posed by a customer is  particularly complex or otherwise cannot be answered at the Tier 1 level,  the call is transferred to one of the two USCIS-operated call centers,  known as Tier 2. Figure 1 shows the organization of NCSC, including the  call centers.
In fiscal year 2004, almost half of 21.1 million calls made to NCSC were  handled and completed by the interactive voice response system and the  rest were generally routed to Tier 1. Customer service representatives  (CSR) at Tier 1 respond to inquiries in English or Spanish. The CSRs focus  primarily on providing administrative information to customers by using a  series of scripts provided by USCIS. For example, if a customer needs  what USCIS considers basic information, such as USCIS local offices’  hours of operations, eligibility requirements, and procedures to follow,  such questions are to be answered by CSRs at Tier 1 call centers using  specific scripts. In addition, CSRs are to refer customers to USCIS service  centers and local offices, for such things as changes of address and  appointment scheduling at USCIS application support centers. (Some of  these tasks may alternatively be performed by customers through the  USCIS Web site—www.uscis.gov.) As of April 2005, the four Tier 1 call  centers employed over 450 CSRs. Figure 2 shows CSRs processing calls at  a Tier 1 call center.
At the two USCIS-operated Tier 2 call centers, calls are handled by  immigration information officers (IIO)—immigration specialists with in- depth knowledge of immigration laws, non-immigrant visas, naturalization,  asylum and refugee status, and other related policies and procedures. As  of April 2005, the Tier 2 call centers operated by USCIS had 111 IIOs.  About 5 percent, or about 590,000, of the calls going to Tier 1 CSRs were  rerouted to Tier 2 IIOs in fiscal year 2004. Figure 3 shows the call volume  handled by the interactive voice response system, Tier 1 call centers, and  Tier 2 call centers during fiscal year 2004.
section_title: Performance-Based Service Contract
paragraphs: In January 2002, USCIS awarded a performance-based service contract for  the management of four Tier 1 call centers. In making this award, USCIS  obtained acquisition services from the Department of Veterans Affairs  (VA), and the contracting officer who signed and was responsible for  administering the contract was a VA employee working on behalf of  USCIS. The contracting officer’s technical representative (COTR), a  USCIS employee, was also responsible for administering the contract. In  commenting on a draft of this report, DHS noted that by agreement of both  the VA and USCIS, on April 20, 2005, USCIS assumed responsibility for  administering the contract. The contract was awarded for a base year,  beginning on June 1, 2002, plus 4 option years (1-year renewable  extensions of the contract, three of which have been exercised as of June  2005). Through calendar year 2004, USCIS paid $64.6 million to the  contractor for the Tier 1 call center operations.
According to the Office of Federal Procurement Policy (OFPP) in the  Office of Management and Budget, performance-based service contracts  are designed to focus on results. Their purpose is to ensure that  contractors are given the freedom to determine how to meet the  government’s performance objectives, that appropriate quality levels are  achieved, and that payment is made only for services that meet these  levels. This type of contract is to emphasize standards for customer  service and measurement of performance and may offer financial  incentives, both positive and negative, to encourage quality performance.  According to OFPP, call centers are suited to this type of contract  because, among other things, they emphasize achieving results by meeting  customer service standards. According to OFPP, with performance-based  service contracts, incentive payments made to an independent contractor  are to be contingent on the contractor’s ability to meet the government’s  performance standards; the contract does not specify how those standards  are to be met. Thus, the contractor retains discretion in determining how  to meet performance standards specified in the contract, for example, how  many CSRs to hire to ensure calls are answered within a contractually  specified time. Other elements suggested for using a performance-based  service contract include (1) identifying the agency’s needs and addressing  those needs with performance requirements that describe required service  results; (2) establishing performance standards that describe the required  performance level; and (3) establishing a quality assurance plan for  assessing contractor performance in order to ensure that the contractor  has performed in accordance with the standards.
section_title: USCIS Established Performance Measures to Assess Contractor’s Call Center Performance
paragraphs: USCIS used a multi-faceted approach to monitor and evaluate the quality  of information and service provided by CSRs to customers calling  contractor-operated Tier 1 call centers. This approach used seven  performance measures. USCIS obtained performance data from the  contractor’s monitoring of selected telephone calls; customer satisfaction  surveys; and a telecommunications vendor (telephone company). In  addition, USCIS used an independent consulting firm to monitor CSRs’  telephone calls and conduct a “mystery shopper” program assessing CSRs’  responses to customers.
subsections: section_title: USCIS Set Call Center Performance Measures for the Contractor to Meet
paragraphs: In order to monitor and evaluate the performance of the four contractor- operated Tier 1 call centers, USCIS planned to use seven performance  measures. These measures were to evaluate the quality of customers’  telephone interactions with CSRs; the accuracy of information provided to  callers over the telephone; the accuracy of callers’ information recorded  by CSRs; callers’ levels of satisfaction; how quickly CSRs handled calls  (two measures); and the number of calls abandoned by customers put on  hold. According to USCIS officials, USCIS established the performance  measures based on a review of industry standards for both government  and private-sector call center operations. The measures were described in  a section of the contract called the Performance Requirements Summary  (PRS).
Under the PRS, these performance measures comprised one of three  components upon which the contractor’s performance score was based.  The other two components were the standard, or goal, set for each  measure, identifying the performance levels the contractor was expected  to meet (e.g., callers will wait an average of 30 to 36 seconds before their  calls are answered), and the performance calculation that USCIS would  use to analyze performance data (e.g., total delay of all calls divided by the  total number of calls). The PRS listing of the seven performance  measures included a “sample calculation” for each of the measures, and  stated that “actual calculations  to be determined during Contract  negotiations.” USCIS officials said they intended to negotiate and finalize  the calculations after a 4- to 6-month phase-in period, and the contract was  awarded with this provisional language.
As to the performance measures and their related standards or goals, three  of USCIS’s performance measures are call quality monitoring, accuracy  of information provided, and accuracy of capturing information. Data  on these measures are to be collected by the contractor’s quality  assurance staff, who are to randomly monitor two calls per day for each  CSR. (CSRs are not to know when they are being monitored.) The data  collected are to be reported to USCIS on a monthly basis. Details on these  three measures follow:    Call quality monitoring. Calls are to be monitored by the contractor’s  quality assurance staff to assess the CSRs’ “soft skills,” that is, their  ability to interact with customers, establish customer rapport, maintain  composure during a call, speak with clarity and professionalism, and  other factors. Call quality monitoring data are to be captured on a  standardized form. CSR responses for each of nine different “soft  skills” are scored as percentages, with scores for the most highly  valued skills, such as “active listening”—that is, whether the CSR was  deemed to be attentive when listening to the customers—given more  weight than the scores for other skills. The nine scores (i.e.,  percentages) are combined for a total “soft skills” score, with 100  percent as the highest possible score. The performance standard stated  in the PRS for this measure is that all calls monitored achieve an  average score of 90 percent to 95 percent after the nine “soft skills”  scores (i.e., percentages) for each call are combined. (See app. II for  additional details on the criteria and methodology used to determine  soft skills scores.)
Accuracy of information provided. Calls are to be monitored by the  contractor’s quality assurance staff to determine, among other things,  whether CSRs provided accurate and complete responses. Using a  standardized form, the staff score CSRs on five different efforts, such  as whether the CSR used software tools appropriately and whether,  when the callers were asked directly, they indicated that their needs  had been satisfied. The five efforts are scored as percentages, with  more weight given to the scores for certain efforts, such as “provides  complete response.” The scores are then combined for a total  “accuracy of information provided” score, with 100 percent as the  highest possible score. The performance standard stated in the PRS for  this measure is that all calls monitored achieve an average score of 95  percent to 97 percent after the five accuracy scores (i.e., percentages)  for each call are combined. (See app. II for additional details on the  criteria and methodology used to determine accuracy of information  provided.)
Accuracy of capturing information. Calls are to be monitored by the  contractor’s quality assurance staff to determine, among other things,  whether CSRs accurately record and verify the callers’ information.  The staff assess this measure by scoring four efforts, including whether  a referral to a local USCIS service center or local office was completed  appropriately and correctly. The four efforts are scored as percentages,  with more weight given to the scores for certain efforts, such as  “verifies caller’s information.” The scores are then combined for a total  “accuracy of information provided” score, with 100 percent as the  highest possible score. The performance standard stated in the PRS for  this measure is that all calls monitored achieve an average score of 95  percent to 97 percent after the four accuracy scores (i.e., percentages)  for each call are combined. (See app. II for additional details on the  criteria and methodology used to determine accuracy of capturing  information.)
A fourth performance measure of call quality—customer satisfaction— was assessed by an independent consulting firm. Customer satisfaction  surveys were conducted on a monthly basis to determine if customers  were satisfied with the service that CSRs provided. At least 375 callers  are to be randomly selected to be interviewed each month from a  population of 10,000 randomly identified callers who called within the 30  days prior to the survey. To measure satisfaction with CSRs, customer  responses to four interview questions about CSRs are compiled, and the  overall percentage of respondents indicating satisfaction is calculated. The  performance standard stated in the PRS for this measure is 80 percent to  85 percent of the customers surveyed indicating overall satisfaction with  the CSRs’ service. (See app. III for additional details on the criteria and  methodology used to determine customer satisfaction.)
Three other performance measures involve the collection of statistical  data by the telecommunications vendor for determining how quickly calls  are answered. The performance measures and standards in the contract  for assessing how quickly CSRs answered customers’ calls are as follows:    Service level. The telecommunications vendor under contract with  USCIS is to collect information on the number of calls answered by  CSRs in 20 seconds or less, that is, the number of callers who spoke to  a CSR within 20 seconds after getting through the interactive voice  response system. The performance standard stated in the PRS for this  measure involves two factors: half-hour increments and the length of  time it took CSRs to answer calls. The standard is that for 80 percent to  85 percent of the half-hour increments measured, 80 percent of the  calls are to be answered in 20 seconds or less.
Average speed of answer. The telecommunications vendor under  contract with USCIS is to collect information on the length of time it  takes for CSRs to answer customers’ calls after they are routed to Tier  1 by the interactive voice response system; that is, how long callers are  on hold before a CSR answers their call. The performance standard  stated in the PRS for this measure is that, for all calls routed to Tier 1,  callers will wait an average of 30 seconds to 36 seconds.
Abandoned calls. The telecommunications vendor under contract with  USCIS is to collect information on the number of calls abandoned by  customers after getting through the interactive voice response system  and waiting for a CSR to answer, that is, the number of times that  customers hang up the telephone while waiting for a CSR. The  performance standard stated in the PRS for this measure involves two  factors: half-hour increments and how frequently callers abandon their  calls. The standard is that for 85 percent to 95 percent of the half-hour  increments measured, 1 percent to 2 percent of the calls are expected  to be abandoned before a CSR answers.
The contract stated that the contractor would be eligible to earn financial  incentive awards if the average monthly performance met or exceeded the  standards on a quarterly basis at each call center, and allowed USCIS to  make deductions from payments to the contractor if the average monthly  performance fell below the standards. According to the contract, the  contractor is not eligible for an incentive award for a particular quarter if  one of the performance standards is not met by one call center, and USCIS  may make a deduction from payments to the contractor in that case. In  addition, USCIS may, at its sole option, elect to include or waive financial  incentives as it deems appropriate.
section_title: USCIS Carried Out Additional Quality Assurance Efforts
paragraphs: In addition to the performance data collected by the contractor’s own  quality assurance staff, an independent consulting firm, and the  telecommunications vendor, USCIS took two additional steps to measure  call center performance for quality assurance purposes. First, to help  ensure that the contractor’s scoring of call-quality performance measures  was reliable, USCIS used another independent consulting firm to validate  the results of the contractor’s efforts by monitoring two calls per month  for each CSR. Data were gathered and provided to USCIS on a monthly  basis. (See app. IV for additional details on the criteria and methodology  used by the independent consulting firm to conduct call monitoring.)
Second, in April 2003, USCIS engaged the same independent consulting  firm to carry out a “mystery shopper” program to assess the completeness  and accuracy of CSRs’ answers to callers. Under this program, an  independent consultant places random calls—1,200 each month—to Tier 1  call centers using various scripts provided by USCIS. As of April 2005, the  scripts used in these calls covered 32 different scenarios, or types of calls,  and 100 new scenarios were being developed. The calls are conducted in  English and Spanish. (See app. V for an example of a mystery shopper  scenario.)
section_title: USCIS Evaluated Contractor’s Performance but Suspended Use of Financial Incentives for More Than 2 Years Due to Performance Measurement Dispute
paragraphs: USCIS did not reach agreement with the contractor on how to apply the  performance measurement requirements described in the PRS before  awarding the performance-based service contract. USCIS suspended all  financial incentives, positive or negative, while the parties negotiated this  issue over a period of about 16 months without reaching agreement. After  negotiations were abandoned, USCIS determined that, for the fourth  quarter of 2004, the contractor had failed to meet four of seven  performance measures and merited a payment deduction. The contractor  disagreed on the grounds that the performance measurements had not  been finalized and that changes in call center workloads affected the basis  for applying financial incentives. In a separate matter, USCIS failed to  ensure that all contractual, regulatory, and GAO standards pertaining to  the documentation of the contractor’s performance were fulfilled.
subsections: section_title: USCIS Awarded the Call Center Contract without Finalizing Performance Measurement Requirements
paragraphs: The performance measurement requirements described in the PRS were  not completely finalized before the contract was awarded. The language  referring to “sample” calculations for determining how performance would  be measured remained in the contract after it was in force. In commenting  on a draft of this report, DHS said that at the time of the contract award,  USCIS management believed it was appropriate to let the winning vendor  have some input into the performance measurement methodology since  this contract represented a transition to performance-based contracting  for call center operations.
The negotiations between USCIS and the contractor on this issue began in  January 2003 (after a phase-in period) and continued intermittently until  April 2004, when they were abandoned. While negotiations were taking  place and after they were abandoned, USCIS obtained monthly data  relating to the contractor’s performance on the seven performance  measures and compared those data to the standards. USCIS considered  the measures and standards themselves to be nonnegotiable; the  contractor, on the other hand, considered them as part of the “sample  calculations” and, thus, negotiable. For over 2 years, USCIS did not use  any of the resulting performance scores for the purpose of calculating  financial incentive awards or payment deductions under the contract  because the terms of the PRS remained unresolved between the parties.
The contractor maintained that the performance scores were “potential  scores” and were to be used by the parties in reaching an agreement on  how to structure the PRS.
On September 1, 2004, the contracting officer, representing USCIS, sent a  letter to the contractor advising that USCIS would begin evaluating the  contractor’s performance and determining a financial incentive award or  payment deduction for the fourth quarter of the calendar year (October 1  through December 31). USCIS officials told us they decided to take this  action because they had concluded that negotiations with the contractor  were unlikely to result in an agreement on the PRS. The contractor  objected to USCIS’s decision to carry out this evaluation. By letter dated  November 29, 2004, the contractor stated that, under the terms of the  contract, USCIS could not unilaterally determine the performance  measurement requirements because all aspects of the requirements were  negotiable, including the performance standards.
The contractor further stated that an evaluation of its performance must  take into account certain changes that took place to the work required  under the contract. For example, the contractor stated that the number of  USCIS-provided scripts, containing information for CSRs to address  callers’ inquiries, had grown to more than 2,300 pages from approximately  400 script pages in June 2002. According to the contractor, these changes  significantly increased the average amount of time needed to handle a call  and affected the contractor’s ability to meet the performance standards  imposed by USCIS. According to the contractor, USCIS’s unilateral  imposition of performance measurement requirements that did not  account for the changed work requirements was inconsistent with Federal  Acquisition Regulation (FAR) 16.402(g), which provides that “t is  essential that the Government and contractor agree explicitly on the effect  that contract changes (e.g., pursuant to the Changes clause) will have on  performance incentives.”
Nevertheless, by letter dated February 11, 2005, USCIS’s contracting  officer notified the contractor of the evaluation results for the period of  October through December 2004. The results showed that the contractor  met the standards for three of the seven performance measures and did  not meet the standards for the other four measures. USCIS determined  that, as a result of this performance, payments due to the contractor for  services would be reduced. The letter noted that the contractor could  submit its own data regarding performance during this period. Following  the review of any data submitted, USCIS would take action to make the  appropriate payment deduction, waive the payment deduction, or pay an  appropriate incentive award.
The contractor requested, by letter dated February 25, 2005, that USCIS  waive implementation of the financial incentives, both positive and  negative. The contractor reiterated its position that USCIS’s unilateral  implementation of the performance measurement requirements as  currently written in the contract, without sufficient regard for substantial  changes to the contract and the changing nature of the program, was not  appropriate. The contractor stated that it was ready to resume  negotiations on this subject so that fair and equitable financial incentives  would be established. The contractor further stated that it had determined  the payment deduction was incorrectly calculated by USCIS.
USCIS’s contracting officer responded, by letter dated April 15, 2005, that  the government would not agree to waive implementation of the financial  incentives and a deduction would be made from the next payment to the  contractor. The letter stated that USCIS did not unilaterally create and  impose the performance measurement requirements, which were included  in the negotiated contract that USCIS and the contractor agreed to.  Regarding the contractor’s assertion that the average amount of time  needed to handle calls had significantly increased, the letter noted that the  performance measurement requirements would apply regardless of the  average length of calls at any given time.
According to FAR and OFPP guidance on performance-based service  contracting, the precise method for measuring performance should have  been agreed upon between USCIS and the contractor before the contract  was signed and implemented. FAR § 16.401 states that performance-based  service contracts should establish “reasonable and attainable   targets that are clearly communicated to the contractor.”  According to OFPP, performance measurement techniques (i.e., how  performance will be assessed to determine whether standards have been  met) are essential elements of performance-based service contracting and  should be clearly stated. In addition, according to OFPP, performance- based service contracts emphasize that all aspects of an acquisition be  structured around the purpose of the work to be performed, that  appropriate performance quality levels are achieved, that payment is made  only for services that meet these levels, and that financial incentives are  awarded to encourage quality performance.
section_title: USCIS Plans to Solicit New Tier 1 Call Center Contracts with Changes to Improve Performance
paragraphs: Although the disagreement between the two parties had not been resolved,  USCIS exercised its option to extend the current call center contract for  another year through May 31, 2006, to allow time to solicit and award new  call center contracts. The exercise of this option has no effect on the  contract’s performance measurement terms, which is the source of the  parties’ dispute. USCIS officials said they plan to award new performance- based service contracts for Tier 1 operations to two vendors, with the two  vendors fully operational by June 2006, to improve the handling of  customers’ calls to Tier 1.
USCIS officials told us they intend for the new contracts to include certain  changes meant to improve Tier 1 call center operations and to incorporate  OFPP guidance on performance-based contracting. USCIS officials told us  that, unlike the current contract, the new PRS will clearly specify how  contractor performance will be assessed and will not leave any terms open  for post-award negotiation. In addition, USCIS officials said the new  contracts will include independent call monitoring and the mystery  shopper program as performance measurement tools to assess the quality  of the Tier 1 CSRs’ responses to customers, including the accuracy and  reliability of the information provided. At the time of our review, USCIS  officials said that the solicitation was going through DHS’s contract review  process and DHS had not issued the solicitation for a new contract  containing these changes. DHS said in its comments on a draft of this  report that the solicitation was with the DHS Procurement Office for  review and issuance.
section_title: USCIS and Contractor Did Not Fully Document Quality Assurance Activities
paragraphs: As part of its quality assurance responsibilities under the current contract,  USCIS is to keep written records of observations about the contractor’s  performance based on periodic evaluations comparing performance data  to standards in the PRS. USCIS’s contracting officer’s technical  representative (COTR), who is responsible for administering the contract,  is to use these written observations to notify the contractor if there are  deficiencies—specifically, if the contractor does not meet the performance  standards. The contractor is required to sign and date such observations to  acknowledge that the COTR apprised it of any deficiencies.
USCIS and contractor officials said they met at least quarterly (monthly,  since October 2004) to discuss performance, performance data, and other  items. USCIS officials said they provided the contractor with  documentation containing performance and other data to discuss at these  meetings. USCIS officials said some of this documentation identified  performance deficiencies. However, contractor officials said they viewed  the performance data as “potential scores” to be considered during  negotiations. To the extent that USCIS considered the performance data as  notification of deficiencies, it did not follow contractual procedures  requiring the COTR to obtain the contractor’s signature acknowledging  notification of the deficiencies. In addition, neither USCIS nor the  contractor kept minutes of these meetings.
According to FAR § 46.104(c), the government should maintain, as part of  the performance records of a contract, suitable records reflecting the  nature of its contract quality assurance actions. With respect to any  performance deficiencies, the government’s records should include,  among other things, the number and type of defects observed and any  actions to correct deficiencies. Further, according to GAO’s standards for  internal control in the federal government, for an agency to run and  control its operations, it must have relevant, reliable information relating  to internal events. All transactions and other significant events need to be  clearly documented, and the documentation should be readily available for  examination. This information should be recorded and communicated to  management and others within the agency who need it to carry out their  responsibilities. GAO’s standards provide a framework for establishing  and maintaining internal control and for identifying and addressing major  performance challenges. Appropriate and effective internal control is a  key factor in helping agencies better achieve program results.
The contract also requires the contractor to provide a quality assurance  plan. The plan that was developed by the contractor describes the  contractor’s approach and strategy for ensuring the delivery of high-quality  service. As part of the plan, the contractor is to conduct formal, biweekly  internal performance review meetings to help with the identification and  correction of performance deficiencies. These meetings are to be attended  by contractor and USCIS officials, with contractor staff reporting on  quality performance issues, and are to be in addition to the quarterly (now  monthly) meetings discussed above. Minutes of the meetings are to  identify action items, responsibilities, and solution time frames, and the  minutes are to be published for USCIS review. However, a contractor  official said that these meetings never took place. According to both  contractor and USCIS officials, the quarterly meetings were used to  discuss operations and performance and to focus senior management  attention on any performance issues.
Under FAR § 37.602-2, the government’s quality assurance surveillance  plans should include actions to help ensure that the contractor carries out  its quality control obligations. By failing to ensure that the contractor held  and documented performance review meetings as required by the  contractor’s quality assurance plan, USCIS did not meet its quality  assurance obligations under FAR § 37.602-2 and GAO’s internal control  standards. In addition, USCIS’s failure to obtain the contractor’s written  acknowledgment of USCIS-identified performance deficiencies did not  meet the notification procedures established by the contract,  documentation requirements of FAR § 46.104(c), and GAO’s standards for  internal control.
In its comments on a draft of this report, DHS noted that the contract was  administered by a component of the VA until April 20, 2005, and DHS said  the VA was provided with documentation discussed at quarterly and  monthly performance assessment meetings between USCIS and the  contractor. According to DHS, the lack of a clear understanding between  USCIS and the VA regarding their roles contributed to the fact that formal  documentation and evaluations were not always properly maintained and  formally transmitted to the contractor. DHS acknowledged that the agency  procuring a service is ultimately responsible for the contract and, thus,  USCIS should have clarified its and the VA’s roles.
section_title: USCIS Has Used Results of Monitoring Efforts to Identify Opportunities to Improve Customer Service and Call Flow at All Call Centers
paragraphs: USCIS used contractor performance data, including the results of surveys,  call monitoring, and the mystery shopper program, to identify  opportunities to improve customer service, including improving call- response times, help CSRs and IIOs better respond to customer inquiries,  and manage the flow of calls into call centers. Following are examples of  initiatives that USCIS recently implemented, or was planning to  implement, as of April 2005. It is too early to assess the impact of these  initiatives.
USCIS implemented “intelligent call routing” for Tier 1. With  “intelligent call routing” in place since February 2004, telephone calls  routed from the interactive voice response system to Tier 1 are now  routed to the next available CSR, at any of the four Tier 1 contractor- operated call centers. Previously, telephone calls to Tier 1 were routed  to the next available CSR in the call center that resided in the same  geographic region of the country as the caller. USCIS officials said that  by June 2005 they will also have implemented “intelligent call routing”  for calls transferred from Tier 1 to the next available IIO at either of the  two Tier 2 call centers.
USCIS implemented “overflow routing.” USCIS started its “overflow  routing” initiative in October 2004, enabling certain general call types,  identified as “English Other” and “Spanish Other,” to be routed directly  from the interactive voice response system to Tier 2 USCIS-operated  call centers, bypassing the Tier 1 contractor-operated call centers.  Previously, all calls handled by IIOs at Tier 2 were first routed from the  interactive voice response system to Tier 1, where CSRs then  transferred the calls to Tier 2. USCIS officials said they expect the  change will result in 1 to 5 percent of all calls being routed directly to  Tier 2, which should help when Tier 1 CSRs cannot handle the call  volume.
USCIS implemented interactive voice response system routing of  certain telephone calls to USCIS service centers. USCIS changed its  automated interactive voice response system in December 2004 so that  certain types of customers’ telephone calls—for example, certain  issues concerning new permanent residents, cases already approved or  denied, and pending cases—are now routed directly to USCIS service  centers, bypassing CSRs at Tier 1. Previously, all customers’ telephone  calls that needed to be handled by USCIS service centers were routed  by the interactive voice response system to Tier 1. Then, after talking  with the customers, the CSRs referred the customers to the service  centers (whose employees have access to case paperwork) via e-mail.  CSRs were allowed to transfer customers’ telephone calls to service  center personnel only when customers requested emergency and  expedited handling of applications.
USCIS implemented a portfolio management system. Private attorneys,  paralegals, and other representatives can use the USCIS Internet Web  site to check the status of their clients’ immigration cases using a  USCIS receipt number. Under the system, USCIS also notifies the  representatives via e-mail when a case status changes; for example,  when actions are taken, such as the approval or denial of an  application. As of April 2005, over 300,000 customers, attorneys, and  other representatives had used this system.
USCIS said it is planning to implement a referral management system.
Currently, Tier 1 CSRs send, via e-mail, service request referrals to  USCIS service centers and local offices for customers who call wanting  to change addresses, schedule and reschedule appointments at  application support centers, order forms, and resolve problems. After a  referral is made, NCSC does not know whether the service center or  local office responded to the customer in a timely manner or even  responded at all. To better monitor this process, USCIS plans to  implement a referral management system, with such service request  referrals placed in a database and assigned a tracking number. The  system is to (1) determine the proper service center or local office to  process the referral, (2) assign the case to an adjudicator, (3) update  the case on a daily basis, and (4) report once a month on case status.  The referral management system is planned to be accessible to  customers on USCIS’s Internet Web site so they can make and track  their own service request referrals. In addition, customers without  Internet access are to be able to call on the telephone and CSR’s will  access the USCIS Web site and create referrals for them. USCIS plans  that the referral management system will be fully operational during  the summer of 2005.
USCIS is planning a customer service portal on USCIS’s Web site.  USCIS has a long-term goal of giving customers Internet access to  information contained in the “scripts” used by Tier 1 CSRs to answer  customers’ questions. USCIS plans to establish a customer service  portal on the USCIS Internet Web site, providing access to the  information. The goal is to let customers with Internet access look up  information themselves without having to call NCSC on the telephone,  navigate the interactive voice response system, and wait for CSR’s to  answer. USCIS had not set a time frame for implementing this  initiative.
section_title: Conclusions
paragraphs: Immigration call centers are a vital information referral source used  millions of times by immigrants and other interested parties seeking to  obtain needed documents, regulatory information, up-to-date status  information on immigration-related benefits and applications, and other  information. To ensure that it serves its customers effectively and  efficiently, USCIS appropriately used a performance-based contract, but  its failure to finalize all aspects of the performance requirements before  the contract was awarded hampered its ability to exercise performance  incentives in the contract. As a result, USCIS lost the opportunity during  the life of the contract to help ensure that it received the maximum level of  service from the contractor.
In addition, USCIS did not meet standards promulgated by federal  acquisition regulations, GAO, and the contract itself pertaining to  documenting the contractor’s performance between 2002 and 2004, and  adequately documenting notification of the contractor when the  government perceived deficiencies in its performance. Failure to generate  adequate documentation could impair USCIS’s ability to conduct future  contract negotiations and to preserve a complete and reliable record of  contract performance needed to ensure accountability.
section_title: Recommendation for Executive Action
paragraphs: To improve USCIS’s efforts for evaluating contractor performance and  encourage quality services at call centers, we recommend that the  Secretary of Homeland Security require the Director of USCIS take the  following two actions: (1) finalize contract terms related to specific  performance measurement requirements before awarding new  performance-based call center contracts; and (2) maintain readily  available written records of performance assessments and performance  evaluation meetings with the contractor.
section_title: Agency Comments and Our Evaluation
paragraphs: DHS and the contractor provided formal comments and technical  comments on a draft of this report, which we have incorporated, as  appropriate. In its formal comments, DHS generally agreed with our  recommendations. DHS said the draft solicitation for the new contracts  specifically identifies performance requirements that are non-negotiable.  DHS further stated that, as recommended by GAO, written records of  performance assessments and performance evaluation meetings will be  maintained and readily available for review by all interested parties. In its  formal comments, the contractor provided additional language to further  clarify this report. The contractor said the report accurately summarizes  the complex nature of CIS’s call center program and several challenges  created by significant post-award changes to that program. DHS’s and the  contractor’s formal comments are shown in appendixes VI and VII,  respectively.
We are sending copies to the Director of USCIS and other interested  parties. We will also make copies available to others upon request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at 202-512-8777 or jonesp@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. Key contributors to this report are listed in  appendix VIII.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine what performance measures the U.S. Citizenship and  Immigration Services (USCIS) established to monitor and evaluate the  performance of contractor-operated call centers, we interviewed USCIS  headquarters officials in Washington, D.C.; Tier 1 contractor officials in  Arlington, Virginia, and at a contractor-operated call center; and an  official representing an independent consulting firm under contract to  USCIS and located in Fairfax, Virginia. We also collected and analyzed  pertinent USCIS and contractor documentation. We collected and  analyzed information on the various types of monitoring and evaluation  programs used by USCIS, including internal call monitoring, independent  call monitoring, customer satisfaction surveys, the mystery shopper  program, and telephone call data provided by a telecommunications  vendor.
To find out how USCIS used the performance measures to evaluate the  contractor’s performance, we interviewed USCIS headquarters officials in  Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia. We  also collected and analyzed pertinent USCIS and contractor  documentation.
To determine what actions, if any, USCIS took or planned to take to  strengthen call center operations, we interviewed USCIS headquarters  officials in Washington, D.C., and Tier 1 contractor officials in Arlington,  Virginia, and at a contractor-operated call center. We also collected and  analyzed pertinent USCIS and contractor documentation.
We assessed the reliability of telephone call volume data provided to  USCIS by a telecommunications vendor, as well as USCIS and contractor  staffing data. To carry out our data reliability assessments, we    (1) reviewed information about the data, systems that produced the data,  and data quality control procedures, and (2) interviewed USCIS and  contractor officials knowledgeable about the data as necessary. We  determined that the call volume and staffing data were sufficiently reliable  for the purposes of this report.
We conducted our work between May 2004 and May 2005 in accordance  with generally accepted auditing standards.
section_title: Appendix II: Criteria and Methodology Used by Contractor to Measure Quality of Tier 1 Calls Monitored
paragraphs: Scoring range   (0 – 3)
Section score  (Percent)
Section score  (Percent)
Scoring range   (0 – 3)
Section score  (Percent)
section_title: Appendix III: Criteria and Methodology Used to Measure Customer Satisfaction with Customer Service Representatives
paragraphs: USCIS and an independent consulting firm jointly developed a telephone  survey to measure customer satisfaction with the three levels of NCSC call  center service—interactive voice response system, Tier 1 CSRs, and Tier 2  IIOs. To carry out the survey each month, representatives of the  independent consulting firm call 375 randomly selected customers. To  assess the customers’ satisfaction with the CSRs, the representatives read  several statements and ask questions for the customers to rate their  experiences with CSRs. For the customer satisfaction performance  measure required in the contract, USCIS collects and summarizes data on  the customers’ responses to the four statements below. The customers are  asked to rate their agreement with each of the statements using a scale of  1 to 7 (1 is strongly agree and 7 is strongly disagree).  1.  The representative seemed to fully understand my questions.  2.  The representative was polite.  3.  The representative did not rush me.  4.  The representative answered my questions promptly.
section_title: Appendix IV: Criteria and Methodology Used by Independent Consulting Firm to Measure Quality of Calls Monitored
paragraphs: An independent consulting firm scored CSRs on 23 separate quality  assurance factors as follows.
subsections: section_title: Service orientation
section_title: Issue identification
section_title: Communications
section_title: Call management
section_title: Appendix V: Mystery Shopper Scenario Example
section_title: Appendix VI: Comments from the Department of Homeland Security
section_title: Appendix VII: Comments from Pearson Government Solutions
section_title: Appendix VIII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contacts
paragraphs: In addition to the above, Darryl W. Dutton, Ronald G. Viereck, Brian J.  Lipman, Christine F. Davis, Amy L. Bernstein, and Michele C. Fejfar made  key contributions to this report.