Report to Congressional Requesters

United States Government Accountability Office

GAO

June 2005 IMMIGRATION 

SERVICES 

Better Contracting 

Practices Needed at 

Call Centers 

GAO-05-526 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

www.gao.gov/cgi-bin/getrpt?GAO-05-526.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Paul Jones at 

(202) 512-8777 or jonespl@gao.gov. 

Highlights of GAO-05-526, a report to

congressional requesters 

June 2005

IMMIGRATION SERVICES 

Better Contracting Practices 

Needed at Call Centers 

USCIS developed seven performance measures intended to assess the 

performance and overall quality of responses provided by customer service 

representatives at contractor-operated call centers. These measures include 

how quickly calls were answered and the accuracy of information provided. 

The contract between USCIS and its contractor stipulated that the 

contractor could earn financial incentive awards if the average monthly 

performance met or exceeded the standards on a quarterly basis at each of 

four call centers. Conversely, financial deductions could be made if the 

standards were not met. 

USCIS did not finalize the terms regarding how the contractor’s actual 

performance would be calculated, or scored, before awarding the contract. 

This limited USCIS’s ability to exercise performance incentives (positive or 

negative) because the parties could not reach agreement on performance 

terms. USCIS suspended the use of financial incentives while the parties 

negotiated the issue. Agreement was not reached after 16 months, however, 

USCIS determined that the contractor had failed to meet standards for 4 of 

the 7 performance measures in the fourth quarter of 2004 and took action to 

reduce its payments for services. The contractor objected, citing the lack of 

agreement on the performance measurements and the impact of workload 

increases, but USCIS disagreed and stated it would reduce payment. In a 

separate but related matter, USCIS failed to meet contractual, regulatory, 

and GAO standards pertaining to how the contractor’s performance would 

be documented—especially with respect to any deficiencies. Finally, USCIS 

exercised its option to extend the call center contract through May 2006, to 

allow time to solicit and award new call center contracts. USCIS said it 

intends to finalize performance measurement terms in the new contracts. 

USCIS used contractor performance data it collected over the course of the 

contract to identify opportunities to improve customer service and call flow, 

among other things. Several initiatives were launched as a result. 

USCIS Call Flow and Call Volume for Fiscal Year 2004 

Source: Developed by GAO from USCIS data.

21.1 million calls

FY 2004

calls

Interactive

Voice 

Response

System 

9.9 million

USCIS

contractor

Automated

system

Call centers

47%

53%

11.2 million

calls

The U.S. Citizenship and 

Immigration Services (USCIS) 

bureau within the Department of 

Homeland Security (DHS) provides 

toll-free telephone assistance 

through call centers to immigrants, 

their attorneys, and others seeking 

information about U.S. immigration 

services and benefits. As the 

volume of calls increased—from 

about 13 million calls in fiscal year 

2002 to about 21 million calls in 

fiscal year 2004—questions were 

raised about USCIS’s ability to 

ensure the reliability and accuracy 

of the information provided at call 

centers run by an independent 

contractor. 

This report analyzes: (1) the 

performance measures established 

by USCIS to monitor and evaluate 

the performance of contractoroperated call centers; (2) how 

performance measures were used 

to evaluate the contractor’s 

performance; and (3) any actions 

USCIS has taken, or plans to take, 

to strengthen call center 

operations. 

What GAO Recommends

To improve USCIS’s evaluation of 

contractor performance, GAO 

recommends that USCIS take steps 

to ensure that performance 

measurement provisions are 

finalized before awarding new 

contracts and that performance 

evaluation records are properly 

maintained. DHS generally agreed 

with GAO’s recommendations and 

indicated USCIS was taking steps 

to implement them. 
Page i GAO-05-526 Immigration Services 

Letter 1

Results in Brief 2 

Background 4 

USCIS Established Performance Measures to Assess Contractor’s 

Call Center Performance 9 

USCIS Evaluated Contractor’s Performance but Suspended Use of 

Financial Incentives for More Than 2 Years Due to Performance 

Measurement Dispute 14 

USCIS Has Used Results of Monitoring Efforts to Identify 

Opportunities to Improve Customer Service and Call Flow at All 

Call Centers 20 

Conclusions 22 

Recommendation for Executive Action 22 

Agency Comments and Our Evaluation 22 

Appendix I Scope and Methodology 25 

Appendix II Criteria and Methodology Used by Contractor to 

Measure Quality of Tier 1 Calls Monitored 27 

Appendix III Criteria and Methodology Used to Measure Customer 

Satisfaction with Customer Service Representatives 28 

Appendix IV Criteria and Methodology Used by Independent 

Consulting Firm to Measure Quality of Calls 

Monitored 29 

Appendix V Mystery Shopper Scenario Example 30 

Appendix VI Comments from the Department of Homeland 

Security 31 

Contents 
Page ii GAO-05-526 Immigration Services 

Appendix VII Comments from Pearson Government Solutions 34 

Appendix VIII GAO Contact and Staff Acknowledgments 36 

Tables 

Table 1: Call Quality Monitoring or “Soft Skills” 27 

Table 2: Accuracy of Information Provided 27 

Table 3: Accuracy of Capturing Information 27 

Figures 

Figure 1: Organization of USCIS’s National Customer Service 

 Center and Call Centers 5 

Figure 2: Tier 1 Customer Service Representatives Processing 

 Immigration-Related Calls at a Contractor-Operated Call 

 Center 6 

Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004 7 
Page iii GAO-05-526 Immigration Services 

Abbreviations 

USCIS U.S. Citizenship and Immigration Services 

COTR contracting officer’s technical representatives 

CSR customer service representative 

DHS Department of Homeland Security 

FAR Federal Acquisition Regulation 

IIO immigration information officer 

INS Immigration and Naturalization Service 

NCSC National Customer Service Center 

OFPP Office of Federal Procurement Policy 

PRS Performance Requirements Summary 

VA Department of Veterans Affairs 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
Page 1 GAO-05-526 Immigration Services 

June 30, 2005 

Congressional Requesters: 

The U.S. Citizenship and Immigration Services (USCIS) bureau within the 

Department of Homeland Security (DHS) provides telephone assistance to 

customers calling about U.S. immigration services and benefits. USCIS 

customers—immigrants and their attorneys or family members, 

employers, and others—can call toll-free for information on such topics as 

how to become U.S. citizens, obtain work visas, initiate the naturalization 

process, obtain legal forms, and determine the status of applications for 

benefits. This telephone assistance service helps customers avoid making 

visits to USCIS offices, which could require customers to take time off 

from work and travel long distances. Telephone calls for assistance to 

USCIS have increased in recent years, from about 13 million in fiscal year 

2002 to about 21 million in fiscal year 2004. These calls are handled in a 

variety of ways: by an interactive voice response system;1

 by four call 

centers managed by a private contractor (Pearson Government Solutions) 

engaged by USCIS; and by two call centers operated directly by USCIS. 

You expressed interest in how USCIS monitors and evaluates contractoroperated call centers to ensure that they operate reliably and provide 

accurate information to their growing number of customers. This report 

addresses the following: (1) What performance measures did USCIS 

establish to monitor and evaluate the performance of contractor-operated 

call centers? (2) How were these performance measures used to evaluate 

the contractor’s performance? (3) What actions, if any, did USCIS take or 

plan to take to strengthen call center operations? 

To address these objectives, we obtained and analyzed information from 

USCIS officials in Washington, D.C., from contractor officials in Arlington, 

Virginia, and at one of four contractor-operated call centers.2

 At these 

locations, we interviewed officials and collected and analyzed pertinent 

documentation, including descriptions of monitoring and evaluation 

programs, contract requirements for call center operations, and 

summaries of call center evaluation results. We assessed the reliability of 

 1

An automated voice system with a touch-tone menu. 

2

The contractor considers the locations of its call centers to be proprietary information. 

United States Government Accountability Office

Washington, DC 20548 
Page 2 GAO-05-526 Immigration Services 

telephone call volume data provided to USCIS by a telecommunications 

vendor, as well as USCIS and contractor staffing data. To carry out our 

data reliability assessments, we (1) reviewed information about the data, 

systems that produced the data, and data quality control procedures, and 

(2) interviewed USCIS and contractor officials knowledgeable about the 

data as necessary. We determined that these data were sufficiently reliable 

for purposes of this report. 

We conducted our work between May 2004 and May 2005 in accordance 

with generally accepted auditing standards. (See app. I for details about 

our scope and methodology.) 

USCIS developed seven performance measures that were intended to 

assess the performance of and overall quality of responses provided by 

customer service representatives at contractor–operated call centers. 

These measures were described in a contract signed by both parties in 

January 2002. Two performance measures, for example, were to assess the 

accuracy of information provided to callers and how quickly calls were 

answered. Based on provisions in the contract, the contractor could earn 

financial incentive awards if certain standards or goals set for each 

performance measure were met or exceeded on a quarterly basis at each 

of the call centers—for instance, if callers waited an average of 30 seconds 

to 36 seconds for their calls to be answered. Conversely, the contract 

provided for negative financial incentives (penalties), whereby USCIS 

could deduct from payments owed to the contractor on a quarterly basis if 

the standards were not met. 

USCIS collected data on the contractor’s performance on a monthly basis 

since June 2002, when the contract was awarded. However, these data 

were not used for the purpose of applying financial incentives (either 

positive or negative) until the fourth quarter of 2004. The use of financial 

incentives was suspended because USCIS awarded the contract without 

finalizing how the contractor’s actual performance would be calculated, or 

scored, for the various performance measures. The two parties negotiated 

on this issue over a period of about 16 months after the contract was 

awarded before abandoning the effort. Thereafter, USCIS determined that, 

for the fourth quarter of 2004, the contractor had failed to meet standards 

for four of the seven performance measures, and announced its intention 

to reduce payments to the contractor for services. The contractor objected 

to USCIS’s decision on the grounds that the performance measurements 

had not been finalized and that changes in call center workloads affected 

the basis for applying financial incentives. USCIS disagreed and stated it 

Results in Brief 
Page 3 GAO-05-526 Immigration Services 

would reduce payment to the contractor. Although this dispute had not 

been resolved, USCIS exercised its option to extend the current call center 

contract through May 2006 to allow time to solicit and award new call 

center contracts. The exercise of this option has no effect on the 

contract’s performance measurement terms, which is the source of the 

parties’ disagreement. In a separate but related matter, USCIS failed to 

meet contractual, regulatory, and GAO standards pertaining to how the 

contractor’s performance was documented—especially with respect to 

any deficiencies. For example, USCIS’s failure to obtain the contractor’s 

written acknowledgement of USCIS-identified performance deficiencies 

did not meet the notification procedures established by the contract, 

documentation requirements of the Federal Acquisition Regulations, and 

GAO’s standards for internal control. Consequently, USCIS did not 

maintain a complete and reliable record of the contractor’s performance 

needed to ensure accountability. 

USCIS used the contractor performance data it collected monthly over the 

course of the contract to identify opportunities to improve customer 

service, help better respond to customer inquiries, and manage the flow of 

calls into call centers. Several initiatives have been launched as a result. 

For example, USCIS has implemented new call-routing procedures 

designed, in part, to route customers’ calls to service representatives more 

quickly and reduce delays (on-hold time) that arise when call volume is 

heavy. These and other initiatives were undertaken by USCIS concurrent 

with its efforts to negotiate contract differences with the contractor. It is 

too early to assess the impact of these initiatives. 

To improve USCIS’s efforts for evaluating contractor performance and 

encourage quality services at call centers, we recommend that the 

Secretary of Homeland Security require the Director of USCIS take the 

following two actions: (1) finalize contract terms related to specific 

performance measurement requirements, before awarding new 

performance-based call center contracts; and (2) maintain readily 

available written records of performance assessments and performance 

evaluation meetings with the contractor. 

DHS and the contractor provided formal comments and technical 

comments on a draft of this report, which we have incorporated, as 

appropriate. DHS generally agreed with our recommendations. The 

contractor said the report accurately summarizes the complex nature of 

CIS’s call center program and several challenges created by significant 

post-award changes to that program. 
Page 4 GAO-05-526 Immigration Services 

The Homeland Security Act of 2002 established USCIS within DHS.3 USCIS 

is responsible for several functions transferred on March 1, 2003, from the 

former Immigration Services Division of the Immigration and 

Naturalization Service (INS) under the Department of Justice.4

 These 

functions include providing services or benefits to facilitate entry, 

residence, employment, and naturalization of legal immigrants; processing 

applications for U.S. citizenship/naturalization; and rendering decisions 

about immigration-related matters. 

The USCIS Information & Customer Service Division is responsible for 

operating the National Customer Service Center (NCSC), which was 

established in 1997 to provide nationwide assistance by telephone to 

customers calling about immigration services and benefits. When a 

customer calls the NCSC toll-free number (1-800-375-5283), the call is 

received by the interactive voice response system. The system features 

automated, self-service options 24-hours a day, 7 days a week. If the 

system cannot address a customer’s concerns or needs or if a customer 

requests live assistance, then the call is generally routed to one of the four 

NCSC contract call centers, known as Tier 1.5

 These four centers are 

operated by the contractor, Pearson. If a question posed by a customer is 

particularly complex or otherwise cannot be answered at the Tier 1 level, 

the call is transferred to one of the two USCIS-operated call centers, 

known as Tier 2.6

 Figure 1 shows the organization of NCSC, including the 

call centers. 

 3

Pub. L. No. 107-296, § 451 (Nov. 25, 2002). 

4

Throughout this report, we use USCIS to refer to both the new USCIS and the former 

Immigration Services Division of INS. 

5

If a customer attempts to obtain live assistance outside normal business hours, the 

customer is told when to call back to obtain the assistance. 

6

In some cases, customers’ calls are routed directly from the interactive voice response 

system to Tier 2, bypassing Tier 1. 

Background 

Call Center Operations 
Page 5 GAO-05-526 Immigration Services 

Figure 1: Organization of USCIS’s National Customer Service Center and Call 

Centers 

In fiscal year 2004, almost half of 21.1 million calls made to NCSC were 

handled and completed by the interactive voice response system and the 

rest were generally routed to Tier 1. Customer service representatives 

(CSR) at Tier 1 respond to inquiries in English or Spanish. The CSRs focus 

primarily on providing administrative information to customers by using a 

series of scripts provided by USCIS. For example, if a customer needs 

what USCIS considers basic information, such as USCIS local offices’ 

hours of operations, eligibility requirements, and procedures to follow, 

such questions are to be answered by CSRs at Tier 1 call centers using 

specific scripts. In addition, CSRs are to refer customers to USCIS service 

centers and local offices, for such things as changes of address and 

appointment scheduling at USCIS application support centers. (Some of 

these tasks may alternatively be performed by customers through the 

USCIS Web site—www.uscis.gov.) As of April 2005, the four Tier 1 call 

Source: Developed by GAO from USCIS data.

NCSC Contractor 

Operations and 

Infrastructure 

Management

- Manages NCSC 

telecommunications 

- Manages NCSC 

contract and monitors 

performance

- Manages NCSC 

quality assurance and 

information provided 

to customers

NCSC Contract Call 

Center Operations

(Tier 1)

NCSC

USCIS Information and 

Customer Service 

Division

Call center 1 Call center 1 --

Los Angeles

Call center 2 --

New York

Call center 2

Call center 3

Call center 4

NCSC Bureau Call 

Center Operations

(Tier 2)
Page 6 GAO-05-526 Immigration Services 

centers employed over 450 CSRs.7 Figure 2 shows CSRs processing calls at 

a Tier 1 call center. 

Figure 2: Tier 1 Customer Service Representatives Processing Immigration-Related Calls at a Contractor-Operated Call 

Center 

At the two USCIS-operated Tier 2 call centers, calls are handled by 

immigration information officers (IIO)—immigration specialists with indepth knowledge of immigration laws, non-immigrant visas, naturalization, 

asylum and refugee status, and other related policies and procedures. As 

of April 2005, the Tier 2 call centers operated by USCIS had 111 IIOs. 

About 5 percent, or about 590,000, of the calls going to Tier 1 CSRs were 

 7

The contractor considers the precise number of CSRs at its call centers and their locations 

to be proprietary information. 

Source: USCIS contractor.
Page 7 GAO-05-526 Immigration Services 

rerouted to Tier 2 IIOs in fiscal year 2004. Figure 3 shows the call volume 

handled by the interactive voice response system, Tier 1 call centers, and 

Tier 2 call centers during fiscal year 2004. 

Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004 

a

“Completed calls” refer to calls that have ended, not necessarily to calls where customers’ questions 

or issues have been resolved. 

In January 2002, USCIS awarded a performance-based service contract for 

the management of four Tier 1 call centers.8

 In making this award, USCIS 

obtained acquisition services from the Department of Veterans Affairs 

(VA), and the contracting officer who signed and was responsible for 

administering the contract was a VA employee working on behalf of 

USCIS.9

 The contracting officer’s technical representative (COTR), a 

USCIS employee, was also responsible for administering the contract. In 

commenting on a draft of this report, DHS noted that by agreement of both 

the VA and USCIS, on April 20, 2005, USCIS assumed responsibility for 

administering the contract. The contract was awarded for a base year, 

 8

The term “contract” in this report refers specifically to “Delivery Order No. 591,” which 

was awarded to the contractor following a competitive procedure through the General 

Services Administration’s Federal Supply Schedule. 

9

 The contract was awarded before DHS was established while what is now USCIS was part 

of INS. 

Performance-Based 

Service Contract 

Source: Developed by GAO from USCIS data.

21.1 million calls

FY 2004

Call

completeda

calls handled 

and completeda

Interactive

Voice 

Response

System 

9.9 million

Automated

system

USCIS call center 

process

Tier 1 CSRs Tier 2 IIOs

10.6 million

calls completeda

0.6 million

calls completeda

If customer selects a touch-tone option to speak with 

a live person, call is routed to Tier 1 CSR.

1 2 12 3 4 47%

53%

Tier 1 CSRs Tier 2 IIOs

11.2 million calls received by Tier 1 CSRs. If 

Tier 1 CSR cannot respond to customer's 

needs, call is forwarded to Tier 2 IIO. 
Page 8 GAO-05-526 Immigration Services 

beginning on June 1, 2002, plus 4 option years (1-year renewable 

extensions of the contract, three of which have been exercised as of June 

2005). Through calendar year 2004, USCIS paid $64.6 million to the 

contractor for the Tier 1 call center operations. 

According to the Office of Federal Procurement Policy (OFPP) in the 

Office of Management and Budget, performance-based service contracts 

are designed to focus on results.10 Their purpose is to ensure that 

contractors are given the freedom to determine how to meet the 

government’s performance objectives, that appropriate quality levels are 

achieved, and that payment is made only for services that meet these 

levels. This type of contract is to emphasize standards for customer 

service and measurement of performance and may offer financial 

incentives, both positive and negative, to encourage quality performance. 

According to OFPP, call centers are suited to this type of contract 

because, among other things, they emphasize achieving results by meeting 

customer service standards. According to OFPP, with performance-based 

service contracts, incentive payments made to an independent contractor 

are to be contingent on the contractor’s ability to meet the government’s 

performance standards; the contract does not specify how those standards 

are to be met. Thus, the contractor retains discretion in determining how 

to meet performance standards specified in the contract, for example, how 

many CSRs to hire to ensure calls are answered within a contractually 

specified time. Other elements suggested for using a performance-based 

service contract include (1) identifying the agency’s needs and addressing 

those needs with performance requirements that describe required service 

results; (2) establishing performance standards that describe the required 

performance level; and (3) establishing a quality assurance plan for 

assessing contractor performance in order to ensure that the contractor 

has performed in accordance with the standards. 

 10OFPP and the National Performance Review coordinated a federal interagency working 

group to develop generic performance-based concepts for government agencies to use in 

the procurement of call center services. The group developed a reference source to help 

program and acquisition officials draft performance requirements and standards related to 

call center performance-based contracts: Performance Based Concepts for Telephone Call 

Center Contracting, Office of Federal Procurement Policy (May 27, 1997). 
Page 9 GAO-05-526 Immigration Services 

USCIS used a multi-faceted approach to monitor and evaluate the quality 

of information and service provided by CSRs to customers calling 

contractor-operated Tier 1 call centers.11 This approach used seven 

performance measures. USCIS obtained performance data from the 

contractor’s monitoring of selected telephone calls; customer satisfaction 

surveys; and a telecommunications vendor (telephone company).12 In 

addition, USCIS used an independent consulting firm to monitor CSRs’ 

telephone calls and conduct a “mystery shopper” program assessing CSRs’ 

responses to customers. 

In order to monitor and evaluate the performance of the four contractoroperated Tier 1 call centers, USCIS planned to use seven performance 

measures. These measures were to evaluate the quality of customers’ 

telephone interactions with CSRs; the accuracy of information provided to 

callers over the telephone; the accuracy of callers’ information recorded 

by CSRs; callers’ levels of satisfaction; how quickly CSRs handled calls 

(two measures); and the number of calls abandoned by customers put on 

hold. According to USCIS officials, USCIS established the performance 

measures based on a review of industry standards for both government 

and private-sector call center operations. The measures were described in 

a section of the contract called the Performance Requirements Summary 

(PRS). 

Under the PRS, these performance measures comprised one of three 

components upon which the contractor’s performance score was based. 

The other two components were the standard, or goal, set for each 

measure, identifying the performance levels the contractor was expected 

to meet (e.g., callers will wait an average of 30 to 36 seconds before their 

calls are answered), and the performance calculation that USCIS would 

use to analyze performance data (e.g., total delay of all calls divided by the 

total number of calls).13 The PRS listing of the seven performance 

measures included a “sample calculation” for each of the measures, and 

 11USCIS monitors and evaluates operations at both Tier 1 and Tier 2 call centers. This 

report focuses on USCIS’s evaluation of Tier 1 contractor-operated call centers. 

12This telecommunications vendor is also responsible for providing the interactive voice 

response system and routing customers’ telephone calls from this system to Tier 1 and Tier 

2 call centers. 

13In this report, we collectively refer to these three components as “performance 

measurement requirements.” 

USCIS Established 

Performance 

Measures to Assess 

Contractor’s Call 

Center Performance 

USCIS Set Call Center 

Performance Measures for 

the Contractor to Meet 
Page 10 GAO-05-526 Immigration Services 

stated that “actual calculations [are] to be determined during Contract 

negotiations.”14 USCIS officials said they intended to negotiate and finalize 

the calculations after a 4- to 6-month phase-in period, and the contract was 

awarded with this provisional language. 

As to the performance measures and their related standards or goals, three 

of USCIS’s performance measures are call quality monitoring, accuracy 

of information provided, and accuracy of capturing information. Data 

on these measures are to be collected by the contractor’s quality 

assurance staff, who are to randomly monitor two calls per day for each 

CSR. (CSRs are not to know when they are being monitored.) The data 

collected are to be reported to USCIS on a monthly basis. Details on these 

three measures follow: 

• Call quality monitoring. Calls are to be monitored by the contractor’s 

quality assurance staff to assess the CSRs’ “soft skills,” that is, their 

ability to interact with customers, establish customer rapport, maintain 

composure during a call, speak with clarity and professionalism, and 

other factors. Call quality monitoring data are to be captured on a 

standardized form. CSR responses for each of nine different “soft 

skills” are scored as percentages, with scores for the most highly 

valued skills, such as “active listening”—that is, whether the CSR was 

deemed to be attentive when listening to the customers—given more 

weight than the scores for other skills. The nine scores (i.e., 

percentages) are combined for a total “soft skills” score, with 100 

percent as the highest possible score. The performance standard stated 

in the PRS for this measure is that all calls monitored achieve an 

average score of 90 percent to 95 percent after the nine “soft skills” 

scores (i.e., percentages) for each call are combined. (See app. II for 

additional details on the criteria and methodology used to determine 

soft skills scores.) 

• Accuracy of information provided. Calls are to be monitored by the 

contractor’s quality assurance staff to determine, among other things, 

whether CSRs provided accurate and complete responses. Using a 

 14The “sample” terms of the contract were never finalized, leading to a dispute between 

USCIS and the contractor that remains unresolved (see p. 15). Examples of “sample 

calculations” listed for different measures in the PRS were “total delay of all calls divided 

by total number of calls” for determining the average length of time it takes before calls are 

answered, and “total number of points achieved divided by total number of points available 

for all quality monitoring scores” for determining call quality. One sample calculation was 

listed for each measure. 
Page 11 GAO-05-526 Immigration Services 

standardized form, the staff score CSRs on five different efforts, such 

as whether the CSR used software tools appropriately and whether, 

when the callers were asked directly, they indicated that their needs 

had been satisfied. The five efforts are scored as percentages, with 

more weight given to the scores for certain efforts, such as “provides 

complete response.” The scores are then combined for a total 

“accuracy of information provided” score, with 100 percent as the 

highest possible score. The performance standard stated in the PRS for 

this measure is that all calls monitored achieve an average score of 95 

percent to 97 percent after the five accuracy scores (i.e., percentages) 

for each call are combined. (See app. II for additional details on the 

criteria and methodology used to determine accuracy of information 

provided.) 

• Accuracy of capturing information. Calls are to be monitored by the 

contractor’s quality assurance staff to determine, among other things, 

whether CSRs accurately record and verify the callers’ information. 

The staff assess this measure by scoring four efforts, including whether 

a referral to a local USCIS service center or local office was completed 

appropriately and correctly. The four efforts are scored as percentages, 

with more weight given to the scores for certain efforts, such as 

“verifies caller’s information.” The scores are then combined for a total 

“accuracy of information provided” score, with 100 percent as the 

highest possible score. The performance standard stated in the PRS for 

this measure is that all calls monitored achieve an average score of 95 

percent to 97 percent after the four accuracy scores (i.e., percentages) 

for each call are combined. (See app. II for additional details on the 

criteria and methodology used to determine accuracy of capturing 

information.) 

A fourth performance measure of call quality—customer satisfaction—

was assessed by an independent consulting firm. Customer satisfaction 

surveys were conducted on a monthly basis to determine if customers 

were satisfied with the service that CSRs provided.15 At least 375 callers 

are to be randomly selected to be interviewed each month from a 

population of 10,000 randomly identified callers who called within the 30 

days prior to the survey. To measure satisfaction with CSRs, customer 

responses to four interview questions about CSRs are compiled, and the 

overall percentage of respondents indicating satisfaction is calculated. The 

 15The customer satisfaction surveys ask about customer satisfaction with the interactive 

voice response system and IIOs, as well as CSRs. 
Page 12 GAO-05-526 Immigration Services 

performance standard stated in the PRS for this measure is 80 percent to 

85 percent of the customers surveyed indicating overall satisfaction with 

the CSRs’ service. (See app. III for additional details on the criteria and 

methodology used to determine customer satisfaction.) 

Three other performance measures involve the collection of statistical 

data by the telecommunications vendor for determining how quickly calls 

are answered. The performance measures and standards in the contract 

for assessing how quickly CSRs answered customers’ calls are as follows: 

• Service level. The telecommunications vendor under contract with 

USCIS is to collect information on the number of calls answered by 

CSRs in 20 seconds or less, that is, the number of callers who spoke to 

a CSR within 20 seconds after getting through the interactive voice 

response system. The performance standard stated in the PRS for this 

measure involves two factors: half-hour increments and the length of 

time it took CSRs to answer calls. The standard is that for 80 percent to 

85 percent of the half-hour increments measured, 80 percent of the 

calls are to be answered in 20 seconds or less. 

• Average speed of answer. The telecommunications vendor under 

contract with USCIS is to collect information on the length of time it 

takes for CSRs to answer customers’ calls after they are routed to Tier 

1 by the interactive voice response system; that is, how long callers are 

on hold before a CSR answers their call. The performance standard 

stated in the PRS for this measure is that, for all calls routed to Tier 1, 

callers will wait an average of 30 seconds to 36 seconds. 

• Abandoned calls. The telecommunications vendor under contract with 

USCIS is to collect information on the number of calls abandoned by 

customers after getting through the interactive voice response system 

and waiting for a CSR to answer, that is, the number of times that 

customers hang up the telephone while waiting for a CSR. The 

performance standard stated in the PRS for this measure involves two 

factors: half-hour increments and how frequently callers abandon their 

calls. The standard is that for 85 percent to 95 percent of the half-hour 

increments measured, 1 percent to 2 percent of the calls are expected 

to be abandoned before a CSR answers. 

The contract stated that the contractor would be eligible to earn financial 

incentive awards if the average monthly performance met or exceeded the 
Page 13 GAO-05-526 Immigration Services 

standards on a quarterly basis at each call center, and allowed USCIS to 

make deductions from payments to the contractor if the average monthly 

performance fell below the standards.16 According to the contract, the 

contractor is not eligible for an incentive award for a particular quarter if 

one of the performance standards is not met by one call center, and USCIS 

may make a deduction from payments to the contractor in that case. In 

addition, USCIS may, at its sole option, elect to include or waive financial 

incentives as it deems appropriate. 

In addition to the performance data collected by the contractor’s own 

quality assurance staff, an independent consulting firm, and the 

telecommunications vendor, USCIS took two additional steps to measure 

call center performance for quality assurance purposes. First, to help 

ensure that the contractor’s scoring of call-quality performance measures 

was reliable, USCIS used another independent consulting firm to validate 

the results of the contractor’s efforts by monitoring two calls per month 

for each CSR. Data were gathered and provided to USCIS on a monthly 

basis. (See app. IV for additional details on the criteria and methodology 

used by the independent consulting firm to conduct call monitoring.) 

Second, in April 2003, USCIS engaged the same independent consulting 

firm to carry out a “mystery shopper” program to assess the completeness 

and accuracy of CSRs’ answers to callers. Under this program, an 

independent consultant places random calls—1,200 each month—to Tier 1 

call centers using various scripts provided by USCIS. As of April 2005, the 

scripts used in these calls covered 32 different scenarios, or types of calls, 

and 100 new scenarios were being developed. The calls are conducted in 

English and Spanish. (See app. V for an example of a mystery shopper 

scenario.) 

 16Up to 10 percent of payments can be added to or deducted from the contractor’s billed 

amounts, depending upon whether or not the performance standards are met. 

USCIS Carried Out 

Additional Quality 

Assurance Efforts 
Page 14 GAO-05-526 Immigration Services 

USCIS did not reach agreement with the contractor on how to apply the 

performance measurement requirements described in the PRS before 

awarding the performance-based service contract. USCIS suspended all 

financial incentives, positive or negative, while the parties negotiated this 

issue over a period of about 16 months without reaching agreement. After 

negotiations were abandoned, USCIS determined that, for the fourth 

quarter of 2004, the contractor had failed to meet four of seven 

performance measures and merited a payment deduction. The contractor 

disagreed on the grounds that the performance measurements had not 

been finalized and that changes in call center workloads affected the basis 

for applying financial incentives. In a separate matter, USCIS failed to 

ensure that all contractual, regulatory, and GAO standards pertaining to 

the documentation of the contractor’s performance were fulfilled. 

The performance measurement requirements described in the PRS were 

not completely finalized before the contract was awarded. The language 

referring to “sample” calculations for determining how performance would 

be measured remained in the contract after it was in force. In commenting 

on a draft of this report, DHS said that at the time of the contract award, 

USCIS management believed it was appropriate to let the winning vendor 

have some input into the performance measurement methodology since 

this contract represented a transition to performance-based contracting 

for call center operations. 

The negotiations between USCIS and the contractor on this issue began in 

January 2003 (after a phase-in period) and continued intermittently until 

April 2004, when they were abandoned. While negotiations were taking 

place and after they were abandoned, USCIS obtained monthly data 

relating to the contractor’s performance on the seven performance 

measures and compared those data to the standards.17 USCIS considered 

the measures and standards themselves to be nonnegotiable; the 

contractor, on the other hand, considered them as part of the “sample 

calculations” and, thus, negotiable. For over 2 years, USCIS did not use 

any of the resulting performance scores for the purpose of calculating 

financial incentive awards or payment deductions under the contract 

because the terms of the PRS remained unresolved between the parties. 

 17USCIS started collecting data on the contractor’s performance in June 2002, the first 

month of the contractor’s operations. We are not presenting the performance data used by 

USCIS because the performance measurement terms of the PRS were unresolved between 

USCIS and the contractor as of April 2005. 

USCIS Evaluated 

Contractor’s 

Performance but 

Suspended Use of 

Financial Incentives 

for More Than 2 Years 

Due to Performance 

Measurement Dispute 

USCIS Awarded the Call 

Center Contract without 

Finalizing Performance 

Measurement 

Requirements 
Page 15 GAO-05-526 Immigration Services 

The contractor maintained that the performance scores were “potential 

scores” and were to be used by the parties in reaching an agreement on 

how to structure the PRS. 

On September 1, 2004, the contracting officer, representing USCIS, sent a 

letter to the contractor advising that USCIS would begin evaluating the 

contractor’s performance and determining a financial incentive award or 

payment deduction for the fourth quarter of the calendar year (October 1 

through December 31). USCIS officials told us they decided to take this 

action because they had concluded that negotiations with the contractor 

were unlikely to result in an agreement on the PRS. The contractor 

objected to USCIS’s decision to carry out this evaluation. By letter dated 

November 29, 2004, the contractor stated that, under the terms of the 

contract, USCIS could not unilaterally determine the performance 

measurement requirements because all aspects of the requirements were 

negotiable, including the performance standards. 

The contractor further stated that an evaluation of its performance must 

take into account certain changes that took place to the work required 

under the contract. For example, the contractor stated that the number of 

USCIS-provided scripts, containing information for CSRs to address 

callers’ inquiries, had grown to more than 2,300 pages from approximately 

400 script pages in June 2002.18 According to the contractor, these changes 

significantly increased the average amount of time needed to handle a call 

and affected the contractor’s ability to meet the performance standards 

imposed by USCIS. According to the contractor, USCIS’s unilateral 

imposition of performance measurement requirements that did not 

account for the changed work requirements was inconsistent with Federal 

Acquisition Regulation (FAR) 16.402(g), which provides that “[i]t is 

essential that the Government and contractor agree explicitly on the effect 

that contract changes (e.g., pursuant to the Changes clause) will have on 

performance incentives.” 

Nevertheless, by letter dated February 11, 2005, USCIS’s contracting 

officer notified the contractor of the evaluation results for the period of 

October through December 2004. The results showed that the contractor 

met the standards for three of the seven performance measures and did 

 18USCIS revised the scripts to provide new and updated immigration-related information on 

a variety of topics. In its technical comments on a draft of this report, DHS said that the 

contractor had been compensated for the increased script content and average call 

duration. 
Page 16 GAO-05-526 Immigration Services 

not meet the standards for the other four measures. USCIS determined 

that, as a result of this performance, payments due to the contractor for 

services would be reduced. The letter noted that the contractor could 

submit its own data regarding performance during this period. Following 

the review of any data submitted, USCIS would take action to make the 

appropriate payment deduction, waive the payment deduction, or pay an 

appropriate incentive award. 

The contractor requested, by letter dated February 25, 2005, that USCIS 

waive implementation of the financial incentives, both positive and 

negative. The contractor reiterated its position that USCIS’s unilateral 

implementation of the performance measurement requirements as 

currently written in the contract, without sufficient regard for substantial 

changes to the contract and the changing nature of the program, was not 

appropriate. The contractor stated that it was ready to resume 

negotiations on this subject so that fair and equitable financial incentives 

would be established. The contractor further stated that it had determined 

the payment deduction was incorrectly calculated by USCIS. 

USCIS’s contracting officer responded, by letter dated April 15, 2005, that 

the government would not agree to waive implementation of the financial 

incentives and a deduction would be made from the next payment to the 

contractor. The letter stated that USCIS did not unilaterally create and 

impose the performance measurement requirements, which were included 

in the negotiated contract that USCIS and the contractor agreed to. 

Regarding the contractor’s assertion that the average amount of time 

needed to handle calls had significantly increased, the letter noted that the 

performance measurement requirements would apply regardless of the 

average length of calls at any given time. 

According to FAR and OFPP guidance19 on performance-based service 

contracting, the precise method for measuring performance should have 

been agreed upon between USCIS and the contractor before the contract 

was signed and implemented. FAR § 16.401 states that performance-based 

service contracts should establish “reasonable and attainable 

[performance] targets that are clearly communicated to the contractor.” 

According to OFPP, performance measurement techniques (i.e., how 

performance will be assessed to determine whether standards have been 

 19Performance Based Concepts for Telephone Call Center Contracting, Office of Federal 

Procurement Policy (May 27, 1997). 
Page 17 GAO-05-526 Immigration Services 

met) are essential elements of performance-based service contracting and 

should be clearly stated. In addition, according to OFPP, performancebased service contracts emphasize that all aspects of an acquisition be 

structured around the purpose of the work to be performed, that 

appropriate performance quality levels are achieved, that payment is made 

only for services that meet these levels, and that financial incentives are 

awarded to encourage quality performance. 

Although the disagreement between the two parties had not been resolved, 

USCIS exercised its option to extend the current call center contract for 

another year through May 31, 2006, to allow time to solicit and award new 

call center contracts.20 The exercise of this option has no effect on the 

contract’s performance measurement terms, which is the source of the 

parties’ dispute. USCIS officials said they plan to award new performancebased service contracts for Tier 1 operations to two vendors, with the two 

vendors fully operational by June 2006, to improve the handling of 

customers’ calls to Tier 1.21

USCIS officials told us they intend for the new contracts to include certain 

changes meant to improve Tier 1 call center operations and to incorporate 

OFPP guidance on performance-based contracting. USCIS officials told us 

that, unlike the current contract, the new PRS will clearly specify how 

contractor performance will be assessed and will not leave any terms open 

for post-award negotiation. In addition, USCIS officials said the new 

contracts will include independent call monitoring and the mystery 

shopper program as performance measurement tools to assess the quality 

of the Tier 1 CSRs’ responses to customers, including the accuracy and 

reliability of the information provided. At the time of our review, USCIS 

officials said that the solicitation was going through DHS’s contract review 

process and DHS had not issued the solicitation for a new contract 

containing these changes. DHS said in its comments on a draft of this 

report that the solicitation was with the DHS Procurement Office for 

review and issuance. 

 20USCIS officials said that DHS will conduct the acquisition and USCIS will administer the 

contract, without seeking inter-agency assistance from the VA as it previously did with the 

current call center contract. 

21The current contractor may compete for the new contracts. 

USCIS Plans to Solicit New 

Tier 1 Call Center 

Contracts with Changes to 

Improve Performance 
Page 18 GAO-05-526 Immigration Services 

As part of its quality assurance responsibilities under the current contract, 

USCIS is to keep written records of observations about the contractor’s 

performance based on periodic evaluations comparing performance data 

to standards in the PRS. USCIS’s contracting officer’s technical 

representative (COTR), who is responsible for administering the contract, 

is to use these written observations to notify the contractor if there are 

deficiencies—specifically, if the contractor does not meet the performance 

standards. The contractor is required to sign and date such observations to 

acknowledge that the COTR apprised it of any deficiencies. 

USCIS and contractor officials said they met at least quarterly (monthly, 

since October 2004) to discuss performance, performance data, and other 

items. USCIS officials said they provided the contractor with 

documentation containing performance and other data to discuss at these 

meetings. USCIS officials said some of this documentation identified 

performance deficiencies. However, contractor officials said they viewed 

the performance data as “potential scores” to be considered during 

negotiations. To the extent that USCIS considered the performance data as 

notification of deficiencies, it did not follow contractual procedures 

requiring the COTR to obtain the contractor’s signature acknowledging 

notification of the deficiencies. In addition, neither USCIS nor the 

contractor kept minutes of these meetings. 

According to FAR § 46.104(c), the government should maintain, as part of 

the performance records of a contract, suitable records reflecting the 

nature of its contract quality assurance actions. With respect to any 

performance deficiencies, the government’s records should include, 

among other things, the number and type of defects observed and any 

actions to correct deficiencies. Further, according to GAO’s standards for 

internal control in the federal government, for an agency to run and 

control its operations, it must have relevant, reliable information relating 

to internal events.22 All transactions and other significant events need to be 

clearly documented, and the documentation should be readily available for 

examination. This information should be recorded and communicated to 

management and others within the agency who need it to carry out their 

responsibilities. GAO’s standards provide a framework for establishing 

and maintaining internal control and for identifying and addressing major 

 22GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999); Internal Control Management and Evaluation Tool, 

GAO-01-1008G (Washington, D.C.: August 2001). 

USCIS and Contractor Did 

Not Fully Document 

Quality Assurance 

Activities 
Page 19 GAO-05-526 Immigration Services 

performance challenges. Appropriate and effective internal control is a 

key factor in helping agencies better achieve program results. 

The contract also requires the contractor to provide a quality assurance 

plan. The plan that was developed by the contractor describes the 

contractor’s approach and strategy for ensuring the delivery of high-quality 

service. As part of the plan, the contractor is to conduct formal, biweekly 

internal performance review meetings to help with the identification and 

correction of performance deficiencies. These meetings are to be attended 

by contractor and USCIS officials, with contractor staff reporting on 

quality performance issues, and are to be in addition to the quarterly (now 

monthly) meetings discussed above. Minutes of the meetings are to 

identify action items, responsibilities, and solution time frames, and the 

minutes are to be published for USCIS review. However, a contractor 

official said that these meetings never took place. According to both 

contractor and USCIS officials, the quarterly meetings were used to 

discuss operations and performance and to focus senior management 

attention on any performance issues. 

Under FAR § 37.602-2, the government’s quality assurance surveillance 

plans should include actions to help ensure that the contractor carries out 

its quality control obligations. By failing to ensure that the contractor held 

and documented performance review meetings as required by the 

contractor’s quality assurance plan, USCIS did not meet its quality 

assurance obligations under FAR § 37.602-2 and GAO’s internal control 

standards. In addition, USCIS’s failure to obtain the contractor’s written 

acknowledgment of USCIS-identified performance deficiencies did not 

meet the notification procedures established by the contract, 

documentation requirements of FAR § 46.104(c), and GAO’s standards for 

internal control. 

In its comments on a draft of this report, DHS noted that the contract was 

administered by a component of the VA until April 20, 2005, and DHS said 

the VA was provided with documentation discussed at quarterly and 

monthly performance assessment meetings between USCIS and the 

contractor. According to DHS, the lack of a clear understanding between 

USCIS and the VA regarding their roles contributed to the fact that formal 

documentation and evaluations were not always properly maintained and 

formally transmitted to the contractor. DHS acknowledged that the agency 

procuring a service is ultimately responsible for the contract and, thus, 

USCIS should have clarified its and the VA’s roles. 
Page 20 GAO-05-526 Immigration Services 

USCIS used contractor performance data, including the results of surveys, 

call monitoring, and the mystery shopper program, to identify 

opportunities to improve customer service, including improving callresponse times, help CSRs and IIOs better respond to customer inquiries, 

and manage the flow of calls into call centers. Following are examples of 

initiatives that USCIS recently implemented, or was planning to 

implement, as of April 2005. It is too early to assess the impact of these 

initiatives. 

• USCIS implemented “intelligent call routing” for Tier 1. With 

“intelligent call routing” in place since February 2004, telephone calls 

routed from the interactive voice response system to Tier 1 are now 

routed to the next available CSR, at any of the four Tier 1 contractoroperated call centers. Previously, telephone calls to Tier 1 were routed 

to the next available CSR in the call center that resided in the same 

geographic region of the country as the caller. USCIS officials said that 

by June 2005 they will also have implemented “intelligent call routing” 

for calls transferred from Tier 1 to the next available IIO at either of the 

two Tier 2 call centers. 

• USCIS implemented “overflow routing.” USCIS started its “overflow 

routing” initiative in October 2004, enabling certain general call types, 

identified as “English Other” and “Spanish Other,” to be routed directly 

from the interactive voice response system to Tier 2 USCIS-operated 

call centers, bypassing the Tier 1 contractor-operated call centers. 

Previously, all calls handled by IIOs at Tier 2 were first routed from the 

interactive voice response system to Tier 1, where CSRs then 

transferred the calls to Tier 2. USCIS officials said they expect the 

change will result in 1 to 5 percent of all calls being routed directly to 

Tier 2, which should help when Tier 1 CSRs cannot handle the call 

volume. 

• USCIS implemented interactive voice response system routing of 

certain telephone calls to USCIS service centers. USCIS changed its 

automated interactive voice response system in December 2004 so that 

certain types of customers’ telephone calls—for example, certain 

issues concerning new permanent residents, cases already approved or 

denied, and pending cases—are now routed directly to USCIS service 

centers, bypassing CSRs at Tier 1. Previously, all customers’ telephone 

calls that needed to be handled by USCIS service centers were routed 

by the interactive voice response system to Tier 1. Then, after talking 

with the customers, the CSRs referred the customers to the service 

centers (whose employees have access to case paperwork) via e-mail. 

CSRs were allowed to transfer customers’ telephone calls to service 

USCIS Has Used 

Results of Monitoring 

Efforts to Identify 

Opportunities to 

Improve Customer 

Service and Call Flow 

at All Call Centers 
Page 21 GAO-05-526 Immigration Services 

center personnel only when customers requested emergency and 

expedited handling of applications. 

• USCIS implemented a portfolio management system. Private attorneys, 

paralegals, and other representatives can use the USCIS Internet Web 

site to check the status of their clients’ immigration cases using a 

USCIS receipt number. Under the system, USCIS also notifies the 

representatives via e-mail when a case status changes; for example, 

when actions are taken, such as the approval or denial of an 

application. As of April 2005, over 300,000 customers, attorneys, and 

other representatives had used this system. 

• USCIS said it is planning to implement a referral management system. 

Currently, Tier 1 CSRs send, via e-mail, service request referrals to 

USCIS service centers and local offices for customers who call wanting 

to change addresses, schedule and reschedule appointments at 

application support centers, order forms, and resolve problems. After a 

referral is made, NCSC does not know whether the service center or 

local office responded to the customer in a timely manner or even 

responded at all. To better monitor this process, USCIS plans to 

implement a referral management system, with such service request 

referrals placed in a database and assigned a tracking number. The 

system is to (1) determine the proper service center or local office to 

process the referral, (2) assign the case to an adjudicator, (3) update 

the case on a daily basis, and (4) report once a month on case status. 

The referral management system is planned to be accessible to 

customers on USCIS’s Internet Web site so they can make and track 

their own service request referrals. In addition, customers without 

Internet access are to be able to call on the telephone and CSR’s will 

access the USCIS Web site and create referrals for them. USCIS plans 

that the referral management system will be fully operational during 

the summer of 2005. 

• USCIS is planning a customer service portal on USCIS’s Web site. 

USCIS has a long-term goal of giving customers Internet access to 

information contained in the “scripts” used by Tier 1 CSRs to answer 

customers’ questions. USCIS plans to establish a customer service 

portal on the USCIS Internet Web site, providing access to the 

information. The goal is to let customers with Internet access look up 

information themselves without having to call NCSC on the telephone, 

navigate the interactive voice response system, and wait for CSR’s to 

answer. USCIS had not set a time frame for implementing this 

initiative. 
Page 22 GAO-05-526 Immigration Services 

Immigration call centers are a vital information referral source used 

millions of times by immigrants and other interested parties seeking to 

obtain needed documents, regulatory information, up-to-date status 

information on immigration-related benefits and applications, and other 

information. To ensure that it serves its customers effectively and 

efficiently, USCIS appropriately used a performance-based contract, but 

its failure to finalize all aspects of the performance requirements before 

the contract was awarded hampered its ability to exercise performance 

incentives in the contract. As a result, USCIS lost the opportunity during 

the life of the contract to help ensure that it received the maximum level of 

service from the contractor. 

In addition, USCIS did not meet standards promulgated by federal 

acquisition regulations, GAO, and the contract itself pertaining to 

documenting the contractor’s performance between 2002 and 2004, and 

adequately documenting notification of the contractor when the 

government perceived deficiencies in its performance. Failure to generate 

adequate documentation could impair USCIS’s ability to conduct future 

contract negotiations and to preserve a complete and reliable record of 

contract performance needed to ensure accountability. 

To improve USCIS’s efforts for evaluating contractor performance and 

encourage quality services at call centers, we recommend that the 

Secretary of Homeland Security require the Director of USCIS take the 

following two actions: (1) finalize contract terms related to specific 

performance measurement requirements before awarding new 

performance-based call center contracts; and (2) maintain readily 

available written records of performance assessments and performance 

evaluation meetings with the contractor. 

DHS and the contractor provided formal comments and technical 

comments on a draft of this report, which we have incorporated, as 

appropriate. In its formal comments, DHS generally agreed with our 

recommendations. DHS said the draft solicitation for the new contracts 

specifically identifies performance requirements that are non-negotiable. 

DHS further stated that, as recommended by GAO, written records of 

performance assessments and performance evaluation meetings will be 

maintained and readily available for review by all interested parties. In its 

formal comments, the contractor provided additional language to further 

clarify this report. The contractor said the report accurately summarizes 

the complex nature of CIS’s call center program and several challenges 

Conclusions 

Recommendation for 

Executive Action 

Agency Comments 

and Our Evaluation 
Page 23 GAO-05-526 Immigration Services 

created by significant post-award changes to that program. DHS’s and the 

contractor’s formal comments are shown in appendixes VI and VII, 

respectively. 

We are sending copies to the Director of USCIS and other interested 

parties. We will also make copies available to others upon request. In 

addition, the report will be available at no charge on GAO’s Web site at 

http://www.gao.gov. 

If you or your staff have any questions regarding this report, please 

contact me at 202-512-8777 or jonesp@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on the 

last page of this report. Key contributors to this report are listed in 

appendix VIII. 

Paul L. Jones, Director 

Homeland Security and Justice Issues 
Page 24 GAO-05-526 Immigration Services 

List of Requesters 

The Honorable John Conyers, Jr. 

Ranking Minority Member 

Committee on the Judiciary 

House of Representatives 

The Honorable Bennie G. Thompson 

Ranking Minority Member 

Committee on Homeland Security 

House of Representatives 

The Honorable Sheila Jackson Lee 

Ranking Minority Member, Subcommittee on Immigration, Border 

 Security, and Claims 

Committee on the Judiciary 

House of Representatives 

The Honorable Loretta Sanchez 

Ranking Minority Member, Subcommittee on Economic Security, 

 Infrastructure Protection, and Cybersecurity 

Committee on Homeland Security 

House of Representatives 

The Honorable Janice D. Schakowsky 

House of Representatives 
Appendix I: Scope and Methodology 

Page 25 GAO-05-526 Immigration Services 

To determine what performance measures the U.S. Citizenship and 

Immigration Services (USCIS) established to monitor and evaluate the 

performance of contractor-operated call centers, we interviewed USCIS 

headquarters officials in Washington, D.C.; Tier 1 contractor officials in 

Arlington, Virginia, and at a contractor-operated call center;1

 and an 

official representing an independent consulting firm under contract to 

USCIS and located in Fairfax, Virginia. We also collected and analyzed 

pertinent USCIS and contractor documentation. We collected and 

analyzed information on the various types of monitoring and evaluation 

programs used by USCIS, including internal call monitoring, independent 

call monitoring, customer satisfaction surveys, the mystery shopper 

program, and telephone call data provided by a telecommunications 

vendor. 

To find out how USCIS used the performance measures to evaluate the 

contractor’s performance, we interviewed USCIS headquarters officials in 

Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia. We 

also collected and analyzed pertinent USCIS and contractor 

documentation. 

To determine what actions, if any, USCIS took or planned to take to 

strengthen call center operations, we interviewed USCIS headquarters 

officials in Washington, D.C., and Tier 1 contractor officials in Arlington, 

Virginia, and at a contractor-operated call center. We also collected and 

analyzed pertinent USCIS and contractor documentation. 

We assessed the reliability of telephone call volume data provided to 

USCIS by a telecommunications vendor, as well as USCIS and contractor 

staffing data. To carry out our data reliability assessments, we 

(1) reviewed information about the data, systems that produced the data, 

and data quality control procedures, and (2) interviewed USCIS and 

contractor officials knowledgeable about the data as necessary. We 

determined that the call volume and staffing data were sufficiently reliable 

for the purposes of this report. 

 1

The contractor considers the locations of its call centers to be proprietary information. 

Appendix I: Scope and Methodology 
Appendix I: Scope and Methodology 

Page 26 GAO-05-526 Immigration Services 

We conducted our work between May 2004 and May 2005 in accordance 

with generally accepted auditing standards. 
Appendix II: Criteria and Methodology Used 

by Contractor to Measure Quality of Tier 1 

Calls Monitored 

Page 27 GAO-05-526 Immigration Services 

Table 1: Call Quality Monitoring or “Soft Skills” 

Criteria 

Scoring range 

 (0 – 3) 

Section score

 (Percent)

Establish rapport 3 11.5

Maintain composure 3 12.0

Conversational style 3 11.5

Active listening 3 12.5

Efficient call flow 3 12.5

Opening 3 10.0

Call on hold 3 10.0

Call transferred 3 10.0

Closing 3 10.0

Percent compliant 100%

Source: USCIS contractor. 

Table 2: Accuracy of Information Provided 

Criteria 

Scoring range 

 (0 – 3) 

Section score

 (Percent)

Uses software tools appropriately 3 9.0

Provides accurate response 3 28.0

Provides complete response 3 27.0

Provides USCIS content only 3 9.0

Satisfies caller’s needs 3 27.0

Percent compliant 100%

Source: USCIS contractor. 

Table 3: Accuracy of Capturing Information 

Criteria 

Scoring range 

 (0 – 3) 

Section score

 (Percent)

Completes referral when 

appropriate 3 12.0

Completes referral record correctly 3 28.0

Captures caller’s information 3 30.0

Verifies caller’s information 3 30.0

Percent compliant 100%

Source: USCIS contractor. 

Appendix II: Criteria and Methodology Used 

by Contractor to Measure Quality of Tier 1 

Calls Monitored 
Appendix III: Criteria and Methodology Used 

to Measure Customer Satisfaction with 

Customer Service Representatives 

Page 28 GAO-05-526 Immigration Services 

USCIS and an independent consulting firm jointly developed a telephone 

survey to measure customer satisfaction with the three levels of NCSC call 

center service—interactive voice response system, Tier 1 CSRs, and Tier 2 

IIOs. To carry out the survey each month, representatives of the 

independent consulting firm call 375 randomly selected customers. To 

assess the customers’ satisfaction with the CSRs, the representatives read 

several statements and ask questions for the customers to rate their 

experiences with CSRs. For the customer satisfaction performance 

measure required in the contract, USCIS collects and summarizes data on 

the customers’ responses to the four statements below. The customers are 

asked to rate their agreement with each of the statements using a scale of 

1 to 7 (1 is strongly agree and 7 is strongly disagree).1

1. The representative seemed to fully understand my questions. 

2. The representative was polite. 

3. The representative did not rush me. 

4. The representative answered my questions promptly. 

 1

The complete scale for the four statements is as follows: 1-Strongly agree; 2-Mostly agree; 

3-Somewhat agree, 4-Mixed, Neither agree nor disagree; 5-Somewhat disagree; 6-Mostly 

disagree; and 7-Strongly disagree. 

Appendix III: Criteria and Methodology Used 

to Measure Customer Satisfaction with 

Customer Service Representatives 
Appendix IV: Criteria and Methodology Used 

by Independent Consulting Firm to Measure 

Quality of Calls Monitored 

Page 29 GAO-05-526 Immigration Services 

An independent consulting firm scored CSRs on 23 separate quality 

assurance factors as follows. 

Greeted customer 

Verified customer 

Established rapport 

Used customer name 

Expressed empathy 

Maintained composure 

Expressed commitment 

Offered additional assistance 

Terminated call appropriately 

Obtained information 

Listed actively 

Took responsibility 

Provided thorough information 

Summarized actions 

Projected enthusiastic tone 

Conveyed confidence 

Spoke with clarity 

Used appropriate language 

Controlled call 

Used time efficiently 

Minimized “dead air” 

Extended hold courtesies 

Transferred call appropriately 

Appendix IV: Criteria and Methodology Used 

by Independent Consulting Firm to Measure 

Quality of Calls Monitored 

Service orientation 

Issue identification 

Issue resolution 

Communications 

Call management 
Appendix V: Mystery Shopper Scenario 

Example 

Page 30 GAO-05-526 Immigration Services 

Source: USCIS. 

Appendix V: Mystery Shopper Scenario 

Example 
Appendix VI: Comments from the Department 

of Homeland Security 

Page 31 GAO-05-526 Immigration Services 

Appendix VI: Comments from the 

Department of Homeland Security 
Appendix VI: Comments from the Department 

of Homeland Security 

Page 32 GAO-05-526 Immigration Services 
Appendix VI: Comments from the Department 

of Homeland Security 

Page 33 GAO-05-526 Immigration Services 
Appendix VII: Comments from Pearson 

Government Solutions 

Page 34 GAO-05-526 Immigration Services 

Appendix VII: Comments from Pearson 

Government Solutions 
Appendix VII: Comments from Pearson 

Government Solutions 

Page 35 GAO-05-526 Immigration Services 
Appendix VIII: GAO Contact and Staff 

Acknowledgments 

Page 36 GAO-05-526 Immigration Services 

Paul L. Jones 202-512-8777 

In addition to the above, Darryl W. Dutton, Ronald G. Viereck, Brian J. 

Lipman, Christine F. Davis, Amy L. Bernstein, and Michele C. Fejfar made 

key contributions to this report. 

Appendix VIII: GAO Contact and Staff 

Acknowledgments 

GAO Contacts 

Staff 

Acknowledgments 

(440318) 
The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER