GAO

United States General Accounting Office

Report to the Subcommittee on the 

Census, Committee on Government 

Reform, House of Representatives

February 2000 2000 CENSUS

New Data Capture 

System Progress and 

Risks

GAO/AIMD-00-61

Page 1 GAO/AIMD-00-61 2000 Census Data Capture System

Contents

Letter 3

Appendixes Appendix I: Briefing on the Decennial Census 18

Appendix II: Comments From the Department of Commerce 99

Figures Figure 1: Remaining DCS 2000 Activities 9

Figure 2: Results of Cluster Accuracy Tests for Optical Mark

Recognition 10

Figure 3: Results of Cluster Accuracy Tests for Optical Character 

Recognition 10

Figure 4: Trends in Defects—All Unresolved PTRs by Week 12

Figure 5: Trends in Defects—Unresolved Severe PTRs by Week 13

Abbreviations

DCS Data Capture System

PTR program trouble report

SEI Software Engineering Institute
Page 2 GAO/AIMD-00-61 2000 Census Data Capture System
Page 3 GAO/AIMD-00-61 2000 Census Data Capture System

United States General Accounting Office

Washington, D.C. 20548

Accounting and Information

Management Division

B-284367 Letr

February 4, 2000

The Honorable Dan Miller

Chairman

The Honorable Carolyn B. Maloney

Ranking Minority Member

Subcommittee on the Census

Committee on Government Reform

House of Representatives

The U.S. Census Bureau is preparing to conduct the 2000 decennial 

census−an event that will significantly influence the lives of every U.S. 

resident and constitutes the largest peacetime mobilization event in our 

nation’s history. In particular, Census will be staffing 1.35 million temporary 

field positions to capture 1.5 billion pages of data from about 119 million 

households. To meet this important and massive challenge, the bureau 

plans to rely extensively on information technology, including its new Data 

Capture System (DCS) 2000. DCS 2000 is to be operated at four bureau data 

capture centers1

 to check in, digitally image, and optically read the data 

handwritten onto census forms and convert these data into files that will be 

transmitted to bureau headquarters for tabulation and analysis. 

At your request, we determined the status and quality of DCS 2000 as well 

as the risks the bureau faces in successfully completing the system. This 

report summarizes the information presented at our December 14, 1999, 

briefing to your office. A copy of our briefing is provided in appendix I. 

Results in Brief The bureau has made considerable progress on DCS 2000. For example, as 

of January 7, 2000, DCS 2000 hardware has been installed at all four data 

capture centers, and 21 of the system’s 23 planned application software 

releases have been completed. Despite this progress, however, the bureau 

faces a formidable challenge in delivering promised DCS 2000 capabilities 

on time because much remains to be accomplished and little time remains 

to accomplish it. In particular, the bureau has extended the system’s 

1

The data capture centers are located in Baltimore, Maryland; Jeffersonville, Indiana; 

Pomona, California; and Phoenix, Arizona.
B-284367

Page 4 GAO/AIMD-00-61 2000 Census Data Capture System

schedule by 4 months due largely to requirements changes. And, before its 

revised completion date of February 25, 2000, the bureau must complete 

many important system development and testing activities, including the 

completion of the final two software releases as well as system acceptance, 

site acceptance, and operational tests. Furthermore, the numbers of yet-tobe-resolved defects for DCS 2000 have yet to show a clear and sustained 

downward trend that is expected as a system begins to mature. 

To expedite the completion of DCS 2000, the bureau and its development 

contractor are following an incremental development and deployment 

strategy. Such a strategy can save time because it can get a system into the 

hands of users faster so that problems can be identified sooner rather than 

later. While the bureau’s implementation of this strategy has introduced 

considerable development and test concurrency, which increases the risk 

of defects being found and corrected independently and thus inconsistent 

system baselines being produced, this risk is being mitigated by the 

development contractor through effective management controls, such as 

project risk management and configuration management.

The bureau and its development contractor have taken other steps to 

ensure the successful delivery of DCS 2000. For example, they have 

implemented a formal risk management program to address risks 

proactively, and they have defined and are following requirements 

management processes to ensure that only those changes that are justified 

on the basis of costs, benefits, and risks are approved and made. Also, they 

have added a test event just prior to the system’s operational date to 

evaluate the system’s capabilities in a true operational setting. Another 

factor strongly in Census’ favor is that its development contractor has been 

independently assessed as having highly effective software development 

capabilities in such important areas as software project planning, tracking 

and oversight, configuration management, software quality management, 

and defect prevention.

Nevertheless, delivering promised DCS 2000 capabilities remains at risk 

because less than 2 months remain before data capture operations are to 

begin, leaving very little room for error. Moreover, many important 

development and test activities remain that will likely reveal more system 

defects and thus compound an already uncertain system maturation 

picture. In discussing this risk with DCS 2000 program officials, they agreed 

that delivering promised system capabilities on time is a risk, and they 

subsequently provided evidence that they have (1) designated this as a high 

risk under the DCS 2000 risk management program and (2) defined and 
B-284367

Page 5 GAO/AIMD-00-61 2000 Census Data Capture System

initiated proactive steps to mitigate the risk and its potential impact on the 

program. 

In commenting on a draft of this report the bureau generally agreed with 

our observations and conclusions and stated that it appreciated our 

insights and contributions.

Background The Constitution requires a decennial census of the population in order to 

reapportion seats in the House of Representatives. Public and private 

decisionmakers also use census data on population counts and social and 

economic characteristics for a variety of purposes. For example, state and 

local redistricting; allocations of government funding; and many planning 

and evaluation activities, such as site selection for new schools, market 

research, and evaluations of local labor markets, rely on decennial census 

data. In addition, the census is the only national source of detailed 

population statistics for small geographic areas, such as towns and school 

districts, and for population groups, such as Native Americans. 

Since 1970, the bureau has used essentially the same methodology to 

conduct the census. It develops an address list of the nation’s housing units 

and delivers census forms to those housing units, requesting that 

occupants mail back the completed forms. Most households are sent a 

short form to complete; however, some are asked to complete a long form.2

The bureau then hires temporary census-takers, known as enumerators, by 

the hundreds of thousands to gather the requested information for each 

nonresponding housing unit. For the 2000 census, some households will 

have the option of completing questionnaires via a bureau-operated tollfree telephone number or a bureau-operated Internet site. 

To conduct the 2000 census, the bureau will rely on 10 key systems. These 

systems will enable the bureau to develop and maintain address lists, maps, 

and geographic reference files; collect census data via the Internet; scan 

and process household-completed paper forms; analyze census data; 

2

The Census short form−three pages with eight questions−will be delivered to 

approximately 83 percent of all housing units. It will ask the respondent to provide 

information for up to six household members including information regarding name, age, 

sex, relationship, and race. The long form−40 pages with 53 questions−will be delivered to 

approximately 17 percent of all housing units. For up to six household members, it will ask 

the same questions as the short form as well as questions on social, economic, financial, and 

physical characteristics. 
B-284367

Page 6 GAO/AIMD-00-61 2000 Census Data Capture System

recruit and support temporary workers; facilitate follow-up surveys; and 

track costs and performance related to taking the census. A simplified 

diagram of the census 2000 systems architecture as well as descriptions of 

each system are provided in appendix I.

An important system for this census will be DCS 2000. The bureau will use 

this system to check in forms and digitally image and optically read the 

data handwritten onto census forms and convert these data into files that 

will be transmitted to bureau headquarters for tabulation and analysis. 

Specific DCS 2000 subsystems include:

• The data verification and receipt subsystem, which, among other things, 

(1) receives the paper census forms and prepares them for imaging and 

(2) identifies respondents so the bureau can identify and follow up with 

nonrespondents.

• The scanning and imaging subsystem, which creates an electronic image 

of the paper form. 

• The optical recognition subsystem, which captures census data from the 

electronic form images. 

• The keying subsystem, which is used to manually input data that cannot 

be satisfactorily read from the paper form. 

• The data preparation function, which formats data from the optical 

recognition and keying subsystems and then sends the data to Census 

Bureau headquarters.

The bureau has contracted with Lockheed Martin-Mission Systems to 

develop, deploy, and maintain DCS 2000. As of November 5, 1999, the 

bureau had obligated $127 million for this contract. The total contract value 

as of September 30, 1999, was $153 million. The bureau has also contracted 

with TRW for space and facility management and for DCS 2000 operation at 

three of the four data capture centers.3

Objectives, Scope, and 

Methodology

Our first objective was to determine DCS 2000’s status and quality, i.e., 

whether the system performs in a way that meets specified requirements 

and satisfies user needs. Our second objective was to assess the risks the 

bureau faces in successfully completing the system.

3

The fourth data capture center, located at Census’ National Processing Center in 

Jeffersonville, Indiana, will be operated by the bureau.
B-284367

Page 7 GAO/AIMD-00-61 2000 Census Data Capture System

To determine the status and quality of DCS 2000, we first reviewed system 

development plans, project management plans, activity plans (test, 

deployment, etc.), project status reports, program management briefings, 

and system functional requirements and specifications. We then compared 

original to revised plans and current plans to status and test reports. We 

also identified how the bureau measured the maturity and quality of DCS 

2000 throughout its development life cycle and analyzed these and other 

measures for indicators of progress and system health, including trends in 

defects and the results of testing activities. We observed the DCS 2000 

operational test at the Baltimore data capture center and attended periodic 

DCS 2000 project review meetings between the bureau and the 

development contractor. 

To determine DCS 2000 risks and risk mitigators, we determined bureau 

and development contractor key management processes (e.g., 

requirements management, configuration management, project planning, 

and project tracking and oversight). We analyzed the results of status and 

quality determinations vis-à-vis DCS 2000 expectations and remaining steps 

to complete the system. We also analyzed plans for completing DCS 2000 

and compared these plans to generally accepted system engineering 

principles.

We interviewed bureau and contractor officials throughout our review, 

briefed them on the results of our work, and incorporated updated 

information they provided into our briefing. We performed our work at the 

Census Bureau’s headquarters in Suitland, Maryland; the DCS 2000 

program office in Lanham, Maryland; bureau and development contractor 

facilities in Bowie, Maryland; and the Census Bureau’s data capture center 

in Baltimore, Maryland, from June 1999 through January 2000, in 

accordance with generally accepted government auditing standards. We 

requested comments on a draft of this report from the Department of 

Commerce. The Secretary of Commerce provided us with written 

comments, which are presented in appendix II, and are discussed in the 

“Agency Comments and Our Evaluation” section of this letter.

Considerable Progress 

Made on DCS 2000 But 

Much Remains to Be 

Done in Little Time

The Census Bureau has made considerable progress on DCS 2000. For 

example, as of January 7, 2000, 21 of 23 planned software releases had been 

completed and 6 of 10 major test events had been completed. Further, 

according to the bureau, all DCS 2000 hardware has been installed at all 

sites. As of January 7, 2000, the bureau was reporting that remaining DCS 

2000 tasks were on target with revised expectations. 
B-284367

Page 8 GAO/AIMD-00-61 2000 Census Data Capture System

Nevertheless, DCS 2000’s expected completion is much later than earlier 

planned and, in fact, perilously close to the date when census 2000 is to 

begin. In February 1999, the bureau expected to complete DCS 2000 by 

October 15, 1999. However, the scheduled completion date has since 

changed to February 25, 2000, which is an extension of over 4 months and 

is less than 2 weeks before data capture operations are to begin on 

March 6, 2000. Under normal circumstances, completion deadlines can be 

changed and some schedule slippage can be tolerated. However, the 

demands of the decennial census necessitate that remaining development, 

testing, and deployment activities be completed in less than 2 months. 

DCS 2000 program officials told us that this extension is due largely to 

system requirement changes. Specifically, from May 15, 1999, through 

October 30, 1999, 63 requirements changes were initiated for DCS 2000. 

And, in October and November 1999, two new software releases were 

added to DCS 2000−one slated for January 19, 2000, and one for 

February 11, 2000. The bureau has no plans for additional requirements 

changes. However, if the results of ongoing and planned operational tests 

show that DCS 2000 does not adequately support production environment 

data capture operations, then more changes are possible. 

In addition to the development and deployment of final software releases, 

significant testing activities—including system, site, and operational tests4

−

remain to be completed before February 25, 2000. These activities are 

illustrated in figure 1.

4

DCS 2000 is currently undergoing three types of major test activities: (1) system acceptance 

testing, which assesses whether the system performs according to specifications, (2) site 

acceptance testing, which assesses whether the system performs correctly at its 

deployment site, and (3) operational testing, which assesses whether the system performs 

as intended when operated on-site by those expected to use it.
B-284367

Page 9 GAO/AIMD-00-61 2000 Census Data Capture System

Figure 1: Remaining DCS 2000 Activities

Results from early site acceptance tests showed that not all performance 

targets were achieved; however, the tests were reported as being 

successful either because the problems were corrected and retested or 

because they were scheduled for correcting and retesting as part of a future 

test event. More recent results show that key performance measures are 

being met. For example, as shown in figures 2 and 3, results from “cluster” 

accuracy tests—which were performed to demonstrate the ability of each 

individual configuration (known as “clusters”) of scanners, workstations, 

and servers to scan census forms and recognize data on those forms—were 

within expected tolerances for errors. 

Key event

System acceptance testing

Release 22

Release 23

OTDR at Phoenix

OTDR at Jeffersonville

Four site operations test

Data Capture

07/01/99 01/28/00

01/19/00

02/11/00

12/06/99 02/04/00

12/20/99 02/04/00

02/22/00 02/25/00

03/06/00 11/30/00

Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1

1999 2000

Legend

Planned progress Actual progress Planned milestone
B-284367

Page 10 GAO/AIMD-00-61 2000 Census Data Capture System

Figure 2: Results of Cluster Accuracy Tests for Optical Mark Recognition

Figure 3: Results of Cluster Accuracy Tests for Optical Character Recognition

0 0.5 1.5 2 1

Pomona

Baltimore

Location

Phoenix

= Tick marks show individual error rate measurements at each location

= White area shows acceptable range of error rate (in percentage).

Key

Error rate (percentage)

0 0.5 1.5 2 1

Pomona

Baltimore

Location

Phoenix

Error rate (percentage)

= Tick marks show individual error rate measurements at each location.

= White area shows acceptable range of error rate (in percentage).

Key
B-284367

Page 11 GAO/AIMD-00-61 2000 Census Data Capture System

Similarly, results from the operational test conducted at the Baltimore, 

Maryland, data capture center identified problems that have since been 

reported as fixed and scheduled to be retested as part of a future 

operational test. An additional operational test was conducted at the 

Baltimore data capture center by an independent research organization to 

assess DCS 2000’s accuracy. Results from this test showed that DCS 2000 

exceeded accuracy goals for both short and long Census forms.

Despite these reported successes, DCS 2000’s readiness is still uncertain 

due to the fact that (1) the numbers of yet-to-be-resolved system defects 

have yet to show a clear and sustained downward trend expected as a 

system begins to mature and (2) the time left between the conclusion of 

test events and the day that DCS 2000 must be operational is less than the 

average time it has been taking to resolve defects. Defects are system 

problems (both hardware and software) that require a resolution (referred 

to by Lockheed Martin as Program Trouble Reports, or PTRs). They can be 

attributed to a failure to meet system specifications or to a requirement 

that was not previously anticipated. As figures 4 and 5 further illustrate, we 

found that the number of unresolved PTRs each week during the last 12 

months does not show a maturing trend, including those defects 

categorized as severe.5

 In addition, as mentioned earlier, many key test 

events, which are major producers of PTRs, have yet to be conducted for 

DCS 2000. Lastly, the time left between conclusion of the last test event, 

scheduled for February 25, 2000, and the day that DCS 2000 must be 

operational is 9 days, whereas it has been taking an average of 16 days to 

resolve all PTRs over the last 6 months and 10 days for severe PTRs. 

5

Two categories of severe defects include emergency−a problem with no 

deliverable/procedural workaround available, which blocks operational use of the product 

or completion of all or some test cases or test phases−and critical−a problem with no 

deliverable/procedural workaround available, which significantly degrades operational use 

or which blocks a top-level test case or test phase from completion.
B-284367

Page 12 GAO/AIMD-00-61 2000 Census Data Capture System

Figure 4: Trends in Defects—All Unresolved PTRs by Week

0

50

100

150

200

250

09/05/199810/05/199811/05/199812/05/199801/05/199902/05/199903/05/199904/05/199905/05/199906/05/199907/05/199908/05/199909/05/199910/05/199911/05/199912/05/199901/05/200002/05/200003/05/200004/05/200005/05/2000

# of PTRs

Unresolved PTRs by week 8-week moving average

DCS 2000

Operations

3/6/00
B-284367

Page 13 GAO/AIMD-00-61 2000 Census Data Capture System

Figure 5: Trends in Defects—Unresolved Severe PTRs by Week

Risk Mitigation Steps 

Are Being Taken to 

Address DCS 2000 

Risks

The bureau and its contractors are employing measures to expedite the 

completion of DCS 2000 and to minimize risks related to its tight schedule 

and other aspects of development and deployment. The bureau and 

Lockheed Martin are following an incremental development and 

deployment strategy. Such a strategy is recognized as a best practice for 

large, complex systems by leading public and private organizations 

because it can get a system into the hands of users faster so that problems 

can be identified sooner rather than later. 

0

2

4

6

8

10

12

09/05/199810/05/199811/05/199812/05/199801/05/199902/05/199903/05/199904/05/199905/05/199906/05/199907/05/199908/05/199909/05/199910/05/199911/05/199912/05/199901/05/200002/05/200003/05/200004/05/200005/05/2000

# of PTRs

Unresolved PTRs by week 8-week moving average

DCS 2000

Operations

3/6/00
B-284367

Page 14 GAO/AIMD-00-61 2000 Census Data Capture System

This approach needs to be carefully managed because it has introduced 

considerable development and test concurrency in the DCS 2000 schedule. 

Such concurrency can increase demands for test resources as well as the 

possibility of defects being found and corrected independently, resulting in 

more than one baseline test configuration. Moreover, if defects surface 

during concurrent testing that are severe enough to require all affected test 

activities to be stopped until the baseline is fixed, the deployment schedule 

could be delayed. We found that these risks are being managed by 

Lockheed Martin through the use of multiple test labs and effective 

configuration management processes.

The bureau and Lockheed Martin are also jointly managing DCS 2000 risks 

through a formal risk management program that includes proactive 

identification of risks and analyses of their impacts and probabilities, 

definition of risk mitigation plans, and tracking and reporting of the plans’ 

implementation and effectiveness. Such programs are also recognized as a 

key practice in government and the private sector. In following the risk 

mitigation program, the bureau and Lockheed Martin have already acted to 

resolve several risks related to DCS 2000 site staffing, daily system 

maintenance, and technical support. For example, they have strengthened 

training programs for on-site DCS 2000 personnel, defined technical 

support roles and responsibilities, and increased central technical support. 

More important, the bureau and Lockheed Martin have also identified the 

tight development schedule as a risk and taken measures to reduce the 

risk. For example, they have

• added staff to manage PTR progress;

• begun holding regular PTR status meetings with sites and other support 

areas to review resolution progress;

• developed a rapid response process that requires a fix or workaround 

within 24 hours;

• established an Engineering Review Board to review all change requests 

to ensure that only changes that are justified on the basis of costs, 

benefits, and risks are approved and made; and

• implemented processes to ensure that all releases with new 

functionality are tested during an operational evaluation.

Furthermore, Census and Lockheed Martin have added another test to the 

DCS 2000 schedule to demonstrate simultaneous operation of all four data 

capture centers. The test, planned for February 22 through February 25, 

will include assessments of DCS 2000’s ability to process workload 
B-284367

Page 15 GAO/AIMD-00-61 2000 Census Data Capture System

equivalent to that expected during actual data capture operations, and it 

will include operational testing of the final DCS 2000 software releases.

Another important risk mitigating factor is that Lockheed Martin-Mission 

Systems has been independently assessed as having highly effective 

software development capabilities in important areas such as software 

project planning, tracking and oversight, and configuration management. 

This assessment was performed using models and methods developed by 

Carnegie Mellon University’s Software Engineering Institute (SEI), which is 

recognized for its expertise in software processes. With DCS 2000 as one of 

three systems included in the evaluation, Lockheed Martin-Mission 

Systems was rated as a “Level 5” (on a scale of 1 to 5), meaning that all 

projects use standard processes for developing and maintaining software 

and that such processes are continually improved through quantitative 

feedback and from piloting innovative ideas and technologies. Having such 

capability should help to reduce risks related not only to developmental 

activities such as project planning and oversight, but all remaining DCS 

2000 development, testing, and deployment activities. 

Conclusions The Census Bureau has made considerable progress in acquiring and 

deploying DCS 2000; however, its prospects for completing this system on 

time are still mixed. In particular, significant system functionality remains 

to be developed, acceptance tested, deployed, and site tested. Further, a 

key measure of system quality−trends in unfixed severe defects−is not yet 

moving in a direction consistent with that of a maturing system. Given that 

a number of significant test events have yet to occur and these are likely to 

uncover more defects, the chances of the defect trend moving in a 

favorable direction soon are uncertain. Moreover, given that the average 

time to fix severe defects (about 10 days) over the last 6 months would 

consume the time between the conclusion of later test events and the day 

that DCS 2000 must be operational, the chances of all specified capabilities 

being ready on time are reduced. 

At the same time, results from system-level tests performed so far are 

showing that key DCS 2000 performance targets are being met. 

Additionally, to meet the challenge of a very demanding and fault-intolerant 

schedule, the bureau has been following a plan that provides for 

development and testing of DCS 2000 in a series of increments. This 

strategy could save time by allowing early system capability to get into the 

hands of end users faster. Lastly, the development contractor has been 

following effective processes for risk management, project planning, 
B-284367

Page 16 GAO/AIMD-00-61 2000 Census Data Capture System

tracking and oversight, configuration management, software quality 

management, and defect prevention−all critical to the successful 

completion of DCS 2000.

Nevertheless, the schedule challenge facing the bureau on DCS 2000, 

particularly in light of where the system currently stands and what remains 

to be accomplished, is formidable. Both the bureau and development 

contractor agree and, accordingly, have initiated proactive and appropriate 

risk management steps, such as strengthening oversight over PTR progress 

and requirements changes. While these steps do not guarantee success, 

they should mitigate the risk and its potential impact on the program.

Agency Comments and 

Our Evaluation

In commenting on a draft of this report, the bureau stated that it generally 

agreed with our observations and conclusions. Additionally, the bureau 

stated that it shared our awareness of the challenges the bureau faces in 

delivering promised DCS 2000 capabilities on time, and that it appreciated 

our insights and contributions. 

The bureau also offered two additional comments. First, it stated that one 

sentence in the draft report implies that all of the DCS 2000 system must be 

developed, tested, and deployed in 2 months while, in fact, the system is in 

the final stages of testing, and only a small portion of development remains 

to be completed before becoming fully operational. It was not our intent to 

imply this and we have accordingly modified the sentence.

Second, the bureau stated that the 16 days that the bureau has averaged 

over the last 6 months to resolve all PTRs that we cite in the draft report is 

not a true representation of the time it has been recently taking to resolve 

emergency PTRs, which according to the bureau was 2 days for 7 recent 

emergency PTRs. We did not modify our report in response to this 

comment. The report does not state that the 16-day average time needed 

over the last 6 months (which covered June through November 1999) to 

resolve all PTRs is the operative indicator of the time to resolve emergency 

PTRs, which is the most severe category of PTRs. Rather, it cites the time 

necessary to resolve severe PTRs—which was 10 days—as the operative 

indicator. This category represents the average time that the bureau’s own 

data show it has taken over the last 6 months to resolve both emergency

and critical PTRs. We focused on both emergency and critical because both 

types of PTRs, according to the Bureau’s own definition of them, will have 

to be resolved before March 6, 2000, if DCS 2000 is to operate as intended.
B-284367

Page 17 GAO/AIMD-00-61 2000 Census Data Capture System

We are sending copies of this report to the Honorable William M. Daley, 

Secretary of Commerce; the Honorable Kenneth Prewitt, Director of the 

U.S. Census Bureau; the Honorable Jacob J. Lew, Director of the Office of 

Management and Budget; and other interested parties. Copies will be made 

available to others upon request.

If you have any questions on matters discussed in this letter, please contact 

me at (202) 512-6240. Other key contributors to this report include Mark 

Bird, Cristina Chaplain, Garry Durfey, Richard Hung, and Aaron Thorne.

Randolph C. Hite

Associate Director

Governmentwide and Defense Information Systems
Page 18 GAO/AIMD-00-61 2000 Census Data Capture System

Appendix I

Briefing on the Decennial Census ApendxiI

1

Accounting and Information Management

Division

Briefing to the

Subcommittee on the Census, Committee on

Government Reform, House of Representatives

Decennial Census: Data Capture System

2000 Progress and Risks

December 14, 1999
Appendix I

Briefing on the Decennial Census

Page 19 GAO/AIMD-00-61 2000 Census Data Capture System

2

Briefing Overview

• Introduction

• Objectives

• Results in Brief

• Background

• Audit Results: DCS 2000 Status and Quality

• Audit Results: DCS 2000 Risks/Risk Mitigators

• Conclusions

• Appendixes

– Scope and Methodology

– Description of Other Decennial Census Systems
Appendix I

Briefing on the Decennial Census

Page 20 GAO/AIMD-00-61 2000 Census Data Capture System

3

Introduction

• Required by the Constitution, our nation’s decennial

census is a landmark event whose results are

instrumental in decisions affecting the lives of every U.S.

citizen.

• The 2000 decennial census is the largest peacetime

mobilization event in our nation’s history, entailing the

staffing of about 1.35 million temporary positions to

collect and capture 1.5 billion pages of data from about

119 million households.

• Information technology is the critical enabler for

successfully accomplishing this mammoth and historical

event.
Appendix I

Briefing on the Decennial Census

Page 21 GAO/AIMD-00-61 2000 Census Data Capture System

4

Objectives

• Because the Data Capture System (DCS) 2000 is

viewed as an important system in accomplishing the

2000 census, the Chairman and Ranking Minority

Member of the Subcommittee asked GAO to determine

the following.

– What is the status and quality of DCS 2000?

– What risks does the bureau face in successfully

completing DCS 2000?

• The scope and methodology of our review is described

in appendix I.
Appendix I

Briefing on the Decennial Census

Page 22 GAO/AIMD-00-61 2000 Census Data Capture System

5

Results in Brief

Status and Quality

• Since February 1999 and largely because of system

requirements changes, the DCS 2000 completion date has

been extended by over 4 months (meaning that the last

system release is now scheduled for deployment less than

4 weeks before the system’s immovable March 6, 2000,

deadline).

• Changes to system requirements continue to occur, with

more activities being added to an already challenging

schedule, but change management controls are in place

and functioning.
Appendix I

Briefing on the Decennial Census

Page 23 GAO/AIMD-00-61 2000 Census Data Capture System

6

Results in Brief (cont.)

• The bureau reports that DCS 2000 is meeting the revised

schedule and that it is meeting cost expectations.

• Trends in yet-to-be resolved system defects are not

consistent with those of a maturing system.

• Five of the 10 scheduled system-level test events have

been completed, and, while early test event results show

that not all performance measures were satisfied, more

recent test event results show that key performance

measures are being met.
Appendix I

Briefing on the Decennial Census

Page 24 GAO/AIMD-00-61 2000 Census Data Capture System

7

Results in Brief (cont.)

Risks

• Important system development, test, and deployment

activities remain that will likely generate more system

defects, and the current schedule will not accommodate

slippage.

• Concurrent system-level test activities increase contention

for test resources and complicate control of test baseline

system configurations.

Risk Mitigators

• Incremental development and deployment gets system

into hands of users faster.
Appendix I

Briefing on the Decennial Census

Page 25 GAO/AIMD-00-61 2000 Census Data Capture System

8

Results in Brief (cont.)

• Formal risk management program is in place to address

risks proactively, including steps to mitigate the possibility

of not completing remaining system development activities

on time.

• An additional test event to exercise all planned system

capabilities in an operational environment is to occur just

prior to the March 6, 2000, deadline.

• Development contractor’s software process capabilities

are effective, including processes for configuration

management.
Appendix I

Briefing on the Decennial Census

Page 26 GAO/AIMD-00-61 2000 Census Data Capture System

9

Results in Brief (cont.)

Bottom Line on DCS 2000 Status, Quality, and Risks

• Considerable progress has been made on DCS 2000 and

important risk mitigators exist.

• Nevertheless, delivering promised system capabilities on

time is a high risk because:

– very little time remains (3 months), and the deadline is

immutable;

– many important development and test activities remain;

– new requirements continue to be added;

– trends in system defects do not yet demonstrate system

maturity; and
Appendix I

Briefing on the Decennial Census

Page 27 GAO/AIMD-00-61 2000 Census Data Capture System

10

Results in Brief (cont.)

– more defects are likely to surface during later test

events.

• On December 7, 1999, we briefed DCS 2000 program

officials on the results of our work and they provided

updated information that has been incorporated into this

briefing as appropriate. The officials agreed that

delivering promised system capabilities on time is a high

risk, and they subsequently provided evidence that

(1) they have designated this as a high risk under the DCS

2000 risk management program and (2) they have defined

and initiated proactive steps to minimize the probability of

the risk occurring and the impact on the program if it does.
Appendix I

Briefing on the Decennial Census

Page 28 GAO/AIMD-00-61 2000 Census Data Capture System

11

Background

Decennial Census Serves Many Purposes

• Public and private decisionmakers use census population

and socioeconomic data for various purposes. For

example,

– the bureau will deliver to the President the population

counts needed to determine each state’s number of

seats in the House of Representatives;

– each state will use tabulations in its individual

redistricting plan;

– federal agencies can use census data to evaluate the

effectiveness of established programs and identify

geographic areas needing attention;
Appendix I

Briefing on the Decennial Census

Page 29 GAO/AIMD-00-61 2000 Census Data Capture System

12

Background

Decennial Census Serves Many Purposes (cont.)

– local governments will use census data to target

delivery of services, such as schools and public

transportation; and

– businesses will use census data to target new services

and products and tailor existing ones to demographic

changes.
Appendix I

Briefing on the Decennial Census

Page 30 GAO/AIMD-00-61 2000 Census Data Capture System

13

Background

Overview of How 2000 Census Data Will Be Captured

• The primary method for gathering census data is through

paper questionnaires mailed to households that the

occupants are to complete and mail back. Paper

questionnaires will also be available in public locations for

people to fill out and mail back.

• Some households will also have the option of responding

via a toll-free telephone number or the Internet, rather than

completing and mailing back paper forms.

• Bureau procedure calls for nonresponding households to

be visited by an enumerator, who will fill out a paper form

for each household.
Appendix I

Briefing on the Decennial Census

Page 31 GAO/AIMD-00-61 2000 Census Data Capture System

14

Background

Overview of Systems Supporting 2000 Census

• The bureau relies on 10 key systems to execute the 2000

decennial census, including DCS 2000. The other nine

systems are listed below and described in appendix II.

– Geographic Support System (GSS)

– Pre-Appointment Management System/Automated

Decennial Administrative Management System

(PAMS/ADAMS)

– Operations Control System (OCS 2000)

– Management Information System (MIS 2000)

– Telephone Questionnaire Assistance and Coverage

Edit Follow-Up (TQA/CEFU)
Appendix I

Briefing on the Decennial Census

Page 32 GAO/AIMD-00-61 2000 Census Data Capture System

15

Background

Overview of Systems Supporting 2000 Census (cont.)

– Internet Data Collection/Internet Questionnaire

Assistance (IDC/IQA)

– Accuracy and Coverage Evaluation System (ACE)

– Headquarters (HQ) Processing

– Data Access and Dissemination System (DADS)
Appendix I

Briefing on the Decennial Census

Page 33 GAO/AIMD-00-61 2000 Census Data Capture System

16

Background

Simplified Diagram of Decennial Census Systems Architecture

Enumerator-completed

paper forms

Census

data

Nonresponse

data

Enumerator assignments

Enumerator

personnel data

Address

listing

Internet

responses IDC/IQA

GSS

MIS 2000

OCS 2000

Telephone

responses

TQA/CEFU

Census

data

users

Headquarters DADS

processing

Adjusted

census data

ACE

Census

data

Census

data

PAMS/

ADAMS

Phoenix DCC

DCS 2000

Pomona DCC

Jeffersonville DCC

Baltimore DCC

Household-completed

paper forms
Appendix I

Briefing on the Decennial Census

Page 34 GAO/AIMD-00-61 2000 Census Data Capture System

17

Background

Simplified Diagram of DCS 2000 Functional

Architecture

Data

preparation

Keying

Data

receipt &

verification

Paper

census

forms Scanning and

imaging

Optical

recognition

Census

data

Census

form

images

Census

data

Census

data

Census

form

images

Unscanned

paper

census

forms

Paper

census

forms

Census

data

DCS 2000

(Key from

paper)

(Key from

image)
Appendix I

Briefing on the Decennial Census

Page 35 GAO/AIMD-00-61 2000 Census Data Capture System

18

Background

DCS 2000 Technical Architecture Overview

• DCS 2000 consists primarily of mail sorters, document

scanners, workstations, and servers running in a

Windows NT-based environment.

• DCS 2000 software is commercial off-the-shelf (COTS);

new software development amounts to about 85,000 lines

of code written in C, C++, and Perl.

• New system development is for modifying COTS, creating

new applications, and integrating components.

• DCS 2000 software is being developed in 23 incremental

releases.

• Scanners, workstations, and servers are configured into

clusters.
Appendix I

Briefing on the Decennial Census

Page 36 GAO/AIMD-00-61 2000 Census Data Capture System

19

Background

DCS 2000 Technical Architecture Overview (cont.)

• Each data capture center (DCC) is configured with the

number of clusters needed to support expected workload.

– Baltimore: 15 clusters

– Jeffersonville: 10 clusters

– Pomona: 14 clusters

– Phoenix: 15 clusters
Appendix I

Briefing on the Decennial Census

Page 37 GAO/AIMD-00-61 2000 Census Data Capture System

20

Cluster LAN

Servers

Scanners

Keying

workstations

Background

DCS 2000 Technical Architecture Overview (cont.)
Appendix I

Briefing on the Decennial Census

Page 38 GAO/AIMD-00-61 2000 Census Data Capture System

21

Background

DCS 2000 Project Structure

• The bureau has contracted with Lockheed Martin-Mission

Systems for DCS 2000 development, deployment, and

maintenance.

• For three of four DCCs, the bureau has contracted with

TRW for space and facility management and for DCS

2000 operation. The bureau plans to operate DCS 2000

at the fourth DCC (the bureau’s National Processing

Center).
Appendix I

Briefing on the Decennial Census

Page 39 GAO/AIMD-00-61 2000 Census Data Capture System

22

Audit Results: DCS 2000 Status and Quality

(cont.)

1. Since February 1999 and largely because of system

requirements changes, the DCS 2000 completion date

has been extended by over 4 months (meaning that the

last system release is now scheduled for deployment

just 4 weeks before the system’s immovable March 6,

2000, deadline).

2. Changes to system requirements continue to occur, with

more activities being added to an already challenging

schedule, but change management controls are in place

and functioning.
Appendix I

Briefing on the Decennial Census

Page 40 GAO/AIMD-00-61 2000 Census Data Capture System

23

Audit Results: DCS 2000 Status and Quality

3. The bureau reports that DCS 2000 is meeting the

revised schedule and that it is meeting cost

expectations.

4. Trends in yet-to-be resolved system defects are not

consistent with those of a maturing system.

5. Five of the 10 scheduled system-level test events have

been completed and while early test event results show

that not all performance measures were satisfied, more

recent test event results show that key performance

measures are being met.
Appendix I

Briefing on the Decennial Census

Page 41 GAO/AIMD-00-61 2000 Census Data Capture System

24

Audit Results: DCS 2000 Status and Quality

1. Scheduled Completion Has Changed

• In February 1999, the scheduled date for completing

DCS 2000 was October 15, 1999. The scheduled date

has since changed to February 25, 2000, an extension of

over 4 months and less than 2 weeks before data capture

operations are to begin.

• Program officials told us that this extension is largely due

to system requirements changes.

• In October/November 1999, major activities were added

to the end of the schedule.

– Two new software releases were added, one on

January 19, 2000, and one on February 11, 2000.
Appendix I

Briefing on the Decennial Census

Page 42 GAO/AIMD-00-61 2000 Census Data Capture System

25

Audit Results: DCS 2000 Status and Quality

1. Scheduled Completion Has Changed (cont.)

– A new operational test at the four DCCs was added for

February 22-25, 2000. This additional event is planned

to demonstrate simultaneous operation of all four

DCCs, including DCS 2000’s capability to process the

workload expected during data capture operation.
Appendix I

Briefing on the Decennial Census

Page 43 GAO/AIMD-00-61 2000 Census Data Capture System

26

Audit Results: DCS 2000 Status and Quality

2. System Requirements Changing But Controlled

• Changes to system requirements, particularly those made

late in the development cycle, can impact the delivery of

promised system capabilities on time and within budget.

Accordingly, changes to requirements must be controlled

to ensure that only changes that are justified on the basis

of costs, benefits, and risks are approved and made.

• For DCS 2000, 63 requirements changes were initiated

between May 15, 1999 and October 30, 1999.

• We reviewed the bureau’s management process for

controlling requirements changes and analyzed selected

DCS 2000 requirements changes to ensure they were

managed in accordance with this management process.
Appendix I

Briefing on the Decennial Census

Page 44 GAO/AIMD-00-61 2000 Census Data Capture System

27

Audit Results: DCS 2000 Status and Quality

2. System Requirements Changing But Controlled (cont.)

• We reviewed five requirements change requests.

Change

control

#006

Submitted

12/15/98

After data have been captured, review all

forms to ensure that headquarters has

received the information.

Reviewed

2/10/99

Approved

Change

control

#028

Submitted

3/1/99

Modify the audit resolution process so that

the system better handles images that fail the

audit process.

Reviewed

12/10/98

Approved

Change

control

#047

Submitted

4/29/99

Create three sets of CD-ROMs of

questionnaire images on a daily basis at

each site for the ACE system.

Reviewed

7/23/99

Approved

Change

control

#070

Submitted

8/2/99

Use remote copy protocol (RCP) to transfer

the Title 13 acknowledgement files from

Census headquarters to the data capture

centers.

Reviewed

8/31/99

Approved

Change

control

#75

Submitted

8/12/99

Add additional checkout reporting data, as

requested by TRW.

Reviewed

9/17/99

Disapproved
Appendix I

Briefing on the Decennial Census

Page 45 GAO/AIMD-00-61 2000 Census Data Capture System

28

Audit Results: DCS 2000 Status and Quality

2. System Requirements Changing But Controlled (cont.)

• We selected these five change requests using the

following criteria: (1) significant in terms of cost or impact,

(2) representative of other change requests, and

(3) demonstrated request approval and disapproval.

Bureau officials agreed that these requests met these

criteria.

• We found that the bureau’s change control management

process provided for reviews of costs, benefits, alternative

solutions, and risks.

• We also found that the five requirements changes that we

reviewed were assessed and approved/disapproved in

accordance with the bureau’s management process.
Appendix I

Briefing on the Decennial Census

Page 46 GAO/AIMD-00-61 2000 Census Data Capture System

29

Audit Results: Status and Quality

3. Progress Against Current Schedule and Cost Baselines

Legend

Planned progress Actual progress Planned milestone Actual milestone

KEY EVENTS

System acceptance test

Release 17

Release 18A

Release 18B

Release 18C

Release 19

Release 20

Site acceptance test at Baltimore

Site acceptance test at Pomona

Site acceptance test at Phoenix

Baltimore operational test

Pomona operational test

07/01/99 01/28/00

06/08/99

06/21/99

07/19/99

08/25/99

10/13/99

11/17/99

07/06/99 08/06/99

09/20/99 10/08/99

11/01/99 11/19/99

08/09/99 10/01/99

10/18/99 11/19/99

May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

1999 2000
Appendix I

Briefing on the Decennial Census

Page 47 GAO/AIMD-00-61 2000 Census Data Capture System

30

Audit Results: DCS 2000 Status and Quality

3. Progress Against Current Schedule and Cost Baselines

(cont.)

• All DCS 2000 hardware reported as installed at all sites.

• Reported DCS 2000 contract costs are as follows:

– Obligated: $127 million reported (as of November 5,

1999) and

– Contract value: $153 million (as of September 30,

1999).
Appendix I

Briefing on the Decennial Census

Page 48 GAO/AIMD-00-61 2000 Census Data Capture System

31

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects Raise Questions About System

Maturity

• One measure of system maturation and quality is trends

in defects.

• Defects are system problems (hardware and software)

that require a resolution. A defect can be due to a failure

to meet the system specifications, or it can occur because

a requirement is identified that was not in the system

specification.

• Effective system engineering requires that defects be

formally managed (identified, documented, and resolved).

To facilitate defect management, defects are categorized

by severity (impact) and age (time unresolved).
Appendix I

Briefing on the Decennial Census

Page 49 GAO/AIMD-00-61 2000 Census Data Capture System

32

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects Raise Questions About Maturity

(cont.)

• Lockheed Martin manages DCS 2000 defects, called

program/trouble reports (PTRs), according to five severity

categories.

Emergency

A problem with no deliverable/procedural workaround

available, which blocks operational use of the product or

completion of all or some test cases or test phases.

Critical

A problem with no deliverable/procedural workaround

available which significantly degrades operational use

or which blocks a top-level test case or test phase from

completion.

Important

A problem with a deliverable/procedural workaround

available (noncritical failed steps in a test case or test

phase would fall here).

Medium A problem discovered through normal use, which is not

covered directly by a test case or test phase.

Low A problem that needs to be fixed but otherwise does not

block development or testing activities.
Appendix I

Briefing on the Decennial Census

Page 50 GAO/AIMD-00-61 2000 Census Data Capture System

33

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects Raise Questions About Maturity

(cont.)

• PTRs are open during the time the software development

organization is designing and implementing a fix or

change. After the development organization has

implemented and tested the fix or change and submitted it

for system integration testing, the PTR is designated fixed.

PTRs are closed after the fix or change has successfully

completed system integration testing.

• We analyzed trends in DCS 2000 PTRs during the last 12

months.
Appendix I

Briefing on the Decennial Census

Page 51 GAO/AIMD-00-61 2000 Census Data Capture System

34

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--All PTRs Opened by Week

-

20

40

60

80

100

120

09/05/199810/05/199811/05/199812/05/1998

01/05/1999

02/05/199903/05/199904/05/199905/05/1999

06/05/1999

07/05/1999

08/05/1999

09/05/1999

10/05/199911/05/199912/05/1999

01/05/2000

02/05/2000

03/05/200004/05/200005/05/2000

# of PTRs

PTRs per week 8-week moving average

Baltimore OTDR

Baltimore SiteAT

System Acceptance Test

Pomona SiteAT

Pomona OTDR

Phoenix SiteAT

Phoenix OTDR

Jeffersonville SiteAT

Jeffersonville OTDR DCS 2000

Operations

3/6/00

Four-site Operations Test
Appendix I

Briefing on the Decennial Census

Page 52 GAO/AIMD-00-61 2000 Census Data Capture System

35

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--Emergency/Critical PTRs

Opened by Week

0

1

2

3

4

5

6

7

8

09/05/199810/05/199811/05/199812/05/1998

01/05/1999

02/05/199903/05/199904/05/199905/05/1999

06/05/1999

07/05/1999

08/05/1999

09/05/199910/05/199911/05/199912/05/199901/05/2000

02/05/2000

03/05/200004/05/200005/05/2000

# of PTRs opened

PTRs per week 8-week moving average

System Acceptance Test

Baltimore SiteAT

Baltimore OTDR

Pomona SiteAT

Pomona OTDR

Phoenix OTDR

Jeffersonville SiteAT

Phoenix SiteAT

Jeffersonville OTDR

DCS 2000

Operations

3/6/00

Four-site Operations Test
Appendix I

Briefing on the Decennial Census

Page 53 GAO/AIMD-00-61 2000 Census Data Capture System

36

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--All Unresolved PTRs by Week

0

50

100

150

200

250

09/05/199810/05/199811/05/199812/05/1998

01/05/199902/05/1999

03/05/199904/05/199905/05/199906/05/199907/05/1999

08/05/1999

09/05/1999

10/05/199911/05/199912/05/199901/05/2000

02/05/2000

03/05/200004/05/200005/05/2000

# of PTRs

Unresolved PTRs by week 8-week moving average

Baltimore OTDR

Baltimore SiteAT

System Acceptance Test

Pomona SiteAT

Pomona OTDR

Phoenix SiteAT

Phoenix OTDR

Jeffersonville SiteAT

Jeffersonville OTDR DCS 2000

Operations

3/6/00

Four-site Operations Test
Appendix I

Briefing on the Decennial Census

Page 54 GAO/AIMD-00-61 2000 Census Data Capture System

37

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--Unresolved Critical/Emergency PTRs

by Week

0

2

4

6

8

10

12

09/05/199810/05/1998

11/05/1998

12/05/199801/05/1999

02/05/199903/05/1999

04/05/199905/05/199906/05/199907/05/1999

08/05/199909/05/199910/05/1999

11/05/199912/05/199901/05/2000

02/05/2000

03/05/200004/05/200005/05/2000

# of PTRs

Unresolved PTRs by week 8-week moving average

System Acceptance Test

Baltimore SiteAT

Baltimore OTDR

Pomona SiteAT

Pomona OTDR

Phoenix OTDR

Jeffersonville SiteAT

Phoenix SiteAT

Jeffersonville OTDR

DCS 2000

Operations

3/6/00

Four-site Operations Test
Appendix I

Briefing on the Decennial Census

Page 55 GAO/AIMD-00-61 2000 Census Data Capture System

38

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--Average Age of All PTRs

0

20

40

60

80

100

120

140

160

180

Nov-98 Dec-98 Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99 Aug-99 Sep-99 Oct-99 Nov-99

Days

Average age of PTRs closed this month Average age of PTRs fixed this month
Appendix I

Briefing on the Decennial Census

Page 56 GAO/AIMD-00-61 2000 Census Data Capture System

39

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--Average Age of Emergency/Critical

PTRs

0

50

100

150

200

250

Nov-98 Dec-98 Jan-99 Feb-99 Mar-99 Apr-99 May-99 Jun-99 Jul-99 Aug-99 Sep-99 Oct-99 Nov-99

Days

Average age of Emergency and Critical PTRs closed this month Average age of Emergency and Critical PTRs fixed this month
Appendix I

Briefing on the Decennial Census

Page 57 GAO/AIMD-00-61 2000 Census Data Capture System

40

Audit Results: DCS 2000 Status and Quality

4. Trends in Defects--Composite View

• Data on the number of PTRs opened and unresolved each

week do not show a maturing trend.

• Test events are major producers of PTRs and many key

test events have yet to occur.

• Average time in the last 6 months to resolve (fix) all PTRs

was about 16 days; emergency and critical PTRs was

about 10 days. A key test event will conclude less than 2

weeks before the system must be operational.
Appendix I

Briefing on the Decennial Census

Page 58 GAO/AIMD-00-61 2000 Census Data Capture System

41

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful

• Acceptance testing is intended to show that a system

correctly performs according to the system specification,

i.e., performs specified functions within defined

performance parameters. Acceptance testing is generally

conducted by the developing organization, with the

acquiring organization’s participation.

• Operational testing is intended to demonstrate that a

system performs as intended when operated on-site by

those expected to use it.

• DCS 2000 is undergoing three types of major test

activities.
Appendix I

Briefing on the Decennial Census

Page 59 GAO/AIMD-00-61 2000 Census Data Capture System

42

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

– System Acceptance Testing (SAT) is to be performed

on the incremental releases of DCS 2000. As of

November 3, 1999, 309 of DCS 2000’s 405 top-level

requirements had been tested (about 76 percent).

– Site Acceptance Testing (SiteAT) is to be performed by

Lockheed Martin, with government witnesses, at each

of the four DCCs on the system functionality released

as of the date of the scheduled SiteAT.

– Baltimore, Pomona, and Phoenix SiteAT are

completed.
Appendix I

Briefing on the Decennial Census

Page 60 GAO/AIMD-00-61 2000 Census Data Capture System

43

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

– According to Lockheed Martin’s test results report for

Baltimore’s SiteAT, the test disclosed various system

problems; however, the test was reported successful

because the problems were either (1) corrected and

retested or (2) scheduled for correction and retesting as

part of a future test event. The following are examples of

tests and results from Baltimore’s SiteAT.
Appendix I

Briefing on the Decennial Census

Page 61 GAO/AIMD-00-61 2000 Census Data Capture System

44

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

Test Name and Description Test Results

Cluster Accuracy : This test’s purpose was

to demonstrate each individual cluster’s

ability to scan a small number of census

forms and recognize data on those forms

within specified tolerances. The expected

tolerance for optical mark recognition

(OMR) errors was 0% to 1.5% and the

expected range for optical character

recognition (OCR) errors was 0% to 2.0%.

The test was reported successful. For

short forms the OMR error rate ranged

from .06% to .66% and the OCR error

rate ranged from .07% to .14%.

Site Accuracy : This test’s purpose was to

demonstrate the accuracy of OMR and

OCR for about 25,000 census form images

on four clusters and create “high

confidence” data files in the format used for

transmission to Census headquarters. The

expected tolerance for OMR errors was 0%

to 1.5% and the expected range for OCR

errors was 0% to 2.0%.

The test was reported successful

although output data files were not

prepared for three of four form types.

OMR and OCR error rates were captured

for all four form types and ranged from

.35% to .94% and between .81% and

1.45%, respectively. Output data files for

the other 3 form types were to be part of

the next SiteAT.
Appendix I

Briefing on the Decennial Census

Page 62 GAO/AIMD-00-61 2000 Census Data Capture System

45

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

Test Name and Description Test Results

Load Stability: This test’s purpose was to

exercise all clusters’ ability to sort and scan

census forms over three 8 hour shifts.

During each shift, 13 clusters’ operations

are simulated while one cluster is operated

by test personnel using paper forms.

Simulations are run at 110 percent of

planned (i.e., normal) operating capacity.

The test was reported successful with

noted exceptions. First, while simulated

sorting of long forms (419,424) exceeded

the goal (360,000), simulated sorting of

short forms failed to meet the expected

goal due to an error in the sorter

simulator. Simulated scanning of both

short (1,205,218) and long (120,787)

forms exceeded the goals of 1,033,116

and 69,888, respectively.

Second, while actual scanning of short

forms exceeded the form per shift goal

(24,596) in two of the three shifts (29,151

and 26,884), it did not in the third

(22,743). Third, scanning of long forms

did not meet the form per shift goal in any

of the three shifts (952, 681, and 1,221

versus a goal of 1,658).
Appendix I

Briefing on the Decennial Census

Page 63 GAO/AIMD-00-61 2000 Census Data Capture System

46

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

– According to Lockheed Martin’s test results report for

Pomona’s SiteAT, the test was reported successful.

– The Pomona SiteAT results were consistent with the

Baltimore SiteAT results for the cluster accuracy and

site accuracy tests and demonstrated improvement for

the load stability test.

– The following are examples of tests and results from

Pomona’s SiteAT.
Appendix I

Briefing on the Decennial Census

Page 64 GAO/AIMD-00-61 2000 Census Data Capture System

47

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

Test Name and Description Test Results

Cluster Accuracy: This test’s purpose was

to demonstrate each individual cluster’s

ability to scan a small number of census

forms and recognize data on those forms

within specified tolerances. The expected

tolerance for optical mark recognition

(OMR) errors was 0% to 1.5% and the

expected range for optical character

recognition (OCR) errors was 0% to 2.0%.

The test was reported successful. For

short forms the OMR error rate ranged

from .0% to .21% and the OCR error rate

ranged from .08% to .28%.

Site Accuracy: This test’s purpose was to

demonstrate the accuracy of OMR and

OCR for 6,000 long census forms on all

clusters. The expected tolerance for OMR

errors was 0% to 1.5% and the expected

range for OCR errors was 0% to 2.0%.

The test was reported successful. OMR

and OCR error rates were .69% and

1.16% respectively.
Appendix I

Briefing on the Decennial Census

Page 65 GAO/AIMD-00-61 2000 Census Data Capture System

48

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

Test Name and Description Test Results

Load Stability: This test’s purpose was to

exercise all clusters’ ability to sort and scan

census forms over three 8 hour shifts.

During each shift, 14 clusters’ operations

are simulated while one cluster is operated

by test personnel using paper forms.

Simulations are run at 110 percent of

planned (i.e., normal) operating capacity.

The test was reported successful.

Simulated sorting of long forms (585,000)

met the goal (585,000) and simulated

sorting of short forms (2,843,400) met the

goal (2,242,500). Simulated scanning of

both short (1,227,240) and long (84,870)

forms exceeded the goals of 1,033,116

and 69,888, respectively.

While actual scanning of short forms

exceeded the form per shift goal (24,598)

in two of the three shifts (28,638 and

28,785), it did not in the third (24,190).

Also, scanning of long forms exceeded

the 1,664 form per shift goal in each of

the three shifts (1,983; 1,922; and 2,145).
Appendix I

Briefing on the Decennial Census

Page 66 GAO/AIMD-00-61 2000 Census Data Capture System

49

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

– Lockheed Martin’s test results report for Phoenix’s

SiteAT was being reviewed and analyzed by the

bureau.

– DCS 2000 program officials told us the Phoenix SiteAT

was successful and preliminary results demonstrated

improvement over the results from Pomona SiteAT.

– The following are examples of tests and results from

Phoenix’s SiteAT as provided by bureau officials.
Appendix I

Briefing on the Decennial Census

Page 67 GAO/AIMD-00-61 2000 Census Data Capture System

50

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

Test Name and Description Test Results

Cluster Accuracy: This test’s purpose was

to demonstrate each individual cluster’s

ability to scan a small number of census

forms and recognize data on those forms

within specified tolerances. The expected

tolerance for optical mark recognition

(OMR) errors was 0% to 1.5% and the

expected range for optical character

recognition (OCR) errors was 0% to 2.0%.

For short forms the OMR error rate

ranged from .0% to .11% and the OCR

error rate ranged from .15% to .28%.

Site Accuracy: This test’s purpose was to

demonstrate the accuracy of OMR and

OCR for 6,000 long census forms on all

clusters. The expected tolerance for OMR

errors was 0% to 1.5% and the expected

range for OCR errors was 0% to 2.0%.

OMR and OCR error rates were .61% and

1.04% respectively.
Appendix I

Briefing on the Decennial Census

Page 68 GAO/AIMD-00-61 2000 Census Data Capture System

51

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

Test Name and Description Test Results

Load Stability: This test’s purpose was to

exercise all clusters’ ability to sort and scan

census forms over three 8 hour shifts.

During each shift, 14 clusters’ operations

are simulated while one cluster is operated

by test personnel using paper forms.

Simulations are run at 110 percent of

planned (i.e., normal) operating capacity.

The results for simulated sorting and

simulated scanning are still being

analyzed.

Actual scanning of short forms exceeded

the form per shift goal (24,598) in each of

the three shifts (32,795; 30,772; and

34,412). Also, scanning of long forms

exceeded the 1,664 form per shift goal in

each of the three shifts (2,389; 2,546; and

2,609).
Appendix I

Briefing on the Decennial Census

Page 69 GAO/AIMD-00-61 2000 Census Data Capture System

52

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

– Operations test and dry runs (OTDRs) are to be

conducted by TRW, and witnessed by the bureau, at

each of the three contractor-operated DCCs. The

bureau is to conduct the OTDR at the Jeffersonville

center.

– TRW’s Baltimore test results report identified the

following problems.

› The workload for performing the “key from image”

function was unevenly distributed.

› The “key from paper” interface was not configured

efficiently, thus limiting productivity.
Appendix I

Briefing on the Decennial Census

Page 70 GAO/AIMD-00-61 2000 Census Data Capture System

53

Audit Results: DCS 2000 Status and Quality

5. Test Events Remain; Completed Events Reported

Successful (cont.)

› The “check-out” bar code scanner often misread the

form identification number.

– Each of these problems is reported as fixed and is to be

retested as part of a future OTDR.

– An additional operational test was conducted at the

Baltimore DCC by an independent research

organization to assess DCS 2000’s accuracy. The test

results showed DCS 2000’s short and long census form

OCR accuracy was 99.4% and 99.3%, respectively,

exceeding the 98% accuracy goal. Further, DCS

2000’s 99.6% OMR accuracy for short and long forms

exceeded the 99% accuracy goal.
Appendix I

Briefing on the Decennial Census

Page 71 GAO/AIMD-00-61 2000 Census Data Capture System

54

Audit Results: DCS 2000 Risks/Risk Mitigators

• Risks

1. Important system development, test, and deployment

activities remain that will likely generate more system

defects, and the current schedule will not

accommodate slippage.

2. Concurrent system-level test activities increase

contention for test resources and complicate control of

test baseline system configurations.

• Risk Mitigators

1. Incremental development and deployment gets system

into hands of users faster.
Appendix I

Briefing on the Decennial Census

Page 72 GAO/AIMD-00-61 2000 Census Data Capture System

55

Audit Results: DCS 2000 Risks/Risk Mitigators

(cont.)

2. Formal risk management program is in place to

address risks proactively, including steps to mitigate

the possibility of not completing remaining system

development activities on time.

3. An additional test event to exercise all planned system

capabilities in an operational environment is to occur

just prior to the March 6, 2000 deadline.

4. Development contractor’s software process capabilities

are effective, including processes for configuration

management.
Appendix I

Briefing on the Decennial Census

Page 73 GAO/AIMD-00-61 2000 Census Data Capture System

56

Audit Results: DCS 2000 Risks

1. Activities Remain/Delays Cannot Be Accommodated

• The bureau must be ready to process census forms by

March 6, 2000, thus leaving only 3 months to complete

DCS 2000.

• The following significant DCS 2000 system development

steps remain:

– development and deployment of three releases,

including fixing known defects;

– completion of system acceptance testing;

– completion of one site acceptance test;

– completion of two operational tests; and

– completion of one operational test at all four sites.
Appendix I

Briefing on the Decennial Census

Page 74 GAO/AIMD-00-61 2000 Census Data Capture System

57

Audit Results: DCS 2000 Risks

1. Activities Remain/Delays Cannot Be Accommodated

(cont.)

• The available time between the last system release and

the DCS 2000 operational date is less than 4 weeks; thus

the current schedule cannot tolerate any delays.
Appendix I

Briefing on the Decennial Census

Page 75 GAO/AIMD-00-61 2000 Census Data Capture System

58

Audit Results: DCS 2000 Risks

1. Activities Remain/Delays Cannot Be Accommodated

(cont.)

Key Event

System acceptance testing

Release 21

Release 22

Release 23

Site acceptance test at Jeffersonville

OTDR at Phoenix

OTDR at Jeffersonville

Four site operations test

Data capture

07/01/99 01/28/00

12/17/99

01/19/00

02/11/00

11/29/99 12/17/99

12/06/99 02/04/00

12/20/99 02/04/00

02/22/00 02/25/00

03/06/00 11/30/00

Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1

1999 2000

Legend

Planned progress Actual progress Planned milestone Actual milestone
Appendix I

Briefing on the Decennial Census

Page 76 GAO/AIMD-00-61 2000 Census Data Capture System

59

Audit Results: DCS 2000 Risks

2. Multiple Test Activities Occurring Concurrently

• Under normal system-level test progression, system

acceptance testing occurs first, followed by site

acceptance testing, and then site operational testing.

While some overlap in the conduct of these test events is

normal, the greater the level of concurrency among the

events, the greater the risk.

• Introducing concurrency of test events is one way of

saving time and thus delivering systems sooner. This

practice has proven successful on systems where no

significant problems are expected during the test events

(e.g., FAA Display Channel Complex-Rehost).

• The DCS 2000 schedule provides for as many as three

system-level test events to occur concurrently.
Appendix I

Briefing on the Decennial Census

Page 77 GAO/AIMD-00-61 2000 Census Data Capture System

60

Audit Results: DCS 2000 Risks

2. Multiple Test Activities Occurring Concurrently

(cont.)

• Test concurrency introduces risks that must be formally

and carefully managed. For example:

– Concurrency will increase contention for test resources,

particularly key staff.

– Concurrency introduces the possibility of defects being

found and corrected independently during the different

test activities, resulting in more than one baseline test

configuration.

– Concurrency can require additional time if severe

defects are found that necessitate stopping all affected

test activities, fixing the baseline, and then restarting

the tests.
Appendix I

Briefing on the Decennial Census

Page 78 GAO/AIMD-00-61 2000 Census Data Capture System

61

Audit Results: DCS 2000 Risk Mitigators

1. Incremental Development and Deployment

• Incremental system development and deployment is a

recognized “best practice” used by leading public and

private organizations to mitigate risks on large, complex

systems.

• As previously described, DCS 2000 is being built,

deployed, and operationally tested incrementally. By

doing so, the bureau is able to learn sooner rather than

later about system problems so that it can better manage

their resolution and better ensure that needed system

capabilities will be ready on time.
Appendix I

Briefing on the Decennial Census

Page 79 GAO/AIMD-00-61 2000 Census Data Capture System

62

Audit Results: DCS 2000 Risk Mitigators

2. Formal Risk Management Program in Place

• Carnegie Mellon University’s Software Engineering

Institute (SEI) recommends a formal risk management

program that includes activities to continually identify,

assess, track, control, and report risks for acquisitions like

DCS 2000.

• The bureau and Lockheed Martin are jointly managing

DCS 2000 risk through a formal DCS 2000 risk

management program that includes proactive

identification of risks and analysis of their impacts and

probabilities, definition of mitigation plans, and tracking

and reporting of the plans’ implementation and

effectiveness.
Appendix I

Briefing on the Decennial Census

Page 80 GAO/AIMD-00-61 2000 Census Data Capture System

63

Audit Results: DCS 2000 Risk Mitigators

2. Formal Risk Management Program in Place (cont.)

• Risk severity is categorized as high, medium, or low.

• As of December 13, 1999, five risks were categorized as

high, meaning that they have a high probability of

occurrence and a high program impact.

Risk Mitigation Steps Status

On-site support

personnel are not

adequately trained.

Lockheed Martin is developing

training materials, scheduling and

conducting training, and will rotate

development staff to the DCCs to

augment site personnel.

Training materials have

been developed and on-site

support personnel are to

receive training as they are

brought onto the program.

Development staff have

been identified to go to the

sites.

Daily maintenance of

system administration

software cannot be

performed at night if

a third shift is added.

Census and Lockheed Martin will

assess alternative ways of

increasing system performance

other than adding a third shift.

Alternatives are being

analyzed. A recommended

solution is expected by

January 7, 2000.
Appendix I

Briefing on the Decennial Census

Page 81 GAO/AIMD-00-61 2000 Census Data Capture System

64

Audit Results: DCS 2000 Risk Mitigators

2. Formal Risk Management Program in Place (cont.)

Risk Mitigation Steps Status

Undefined technical

support roles and

responsibilities may

affect site operations.

Lockheed Martin is defining the

roles and responsibilities for the

site operations team and

developing procedures and

guides for use during operations.

The procedures and guides are to

be used during the final 2 OTDRs

and the 4-site operations test.

Roles and responsibilities

are defined in the on-site

support procedures and

guides, which were

completed on December

10, 1999.

Central technical

support staffing levels

are currently

insufficient to provide

technical support

coverage during all

processing shifts at

the DCCs.

Lockheed Martin is seeking a

contract modification to increase

central technical support staffing.

Lockheed Martin is

developing a proposal and

associated cost estimate for

increased central technical

support. The proposal and

cost estimate is expected to

be provided to the Bureau

in January 2000.
Appendix I

Briefing on the Decennial Census

Page 82 GAO/AIMD-00-61 2000 Census Data Capture System

65

Audit Results: DCS 2000 Risk Mitigators

2. Formal Risk Management Program in Place (cont.)

Risk Mitigation Steps Status

Tight development

schedules poses risk

to successful data

capture.

1) Extend release management

function. 2) Increase focus on

PTRs to ensure proper priority

and planning for the remaining

schedule. Review PTR status

regularly and establish

contingency plans for all PTRs,

which may miss committed

schedules. 3) Continue to review

all new change requests to

ensure they are high priority and

that they can be delivered with

manageable risk. 4) Make sure

that all releases with new

functionality are tested during an

OTDR. 5) Continue to develop

and refine the Rapid Response

process, which requires a fix or a

workaround within 24 hours.

1) Proposal submitted and

additional release

management support

authorized. 2) Added staff

to manage PTR progress.

Regular PTR status

meetings held to review

PTRs with sites and other

support areas.

3) Established Engineering

Review Board to review all

change requests.

4) Current release plan

reflects all new functionality

as part of Release 22 and

will be tested during the

Phoenix and Jeffersonville

OTDRs. Release 23 will be

part of 4-site test and will

contain PTRs only.

5) Rapid response

process developed.
Appendix I

Briefing on the Decennial Census

Page 83 GAO/AIMD-00-61 2000 Census Data Capture System

66

Audit Results: DCS 2000 Risk Mitigators

3. Additional Test of All System Capabilities Planned

• A test to demonstrate simultaneous operation of all four

DCC’s, including DCS 2000’s capability to process

workload equivalent to that expected during actual data

capture operation, is planned for February 22-25, 2000.

• The test will include operational testing of all DCS 2000

software releases.

• The test will use the same system operations and support

personnel that will be on board for actual data capture

operations.
Appendix I

Briefing on the Decennial Census

Page 84 GAO/AIMD-00-61 2000 Census Data Capture System

67

Audit Results: DCS 2000 Risk Mitigators

4. Development Contractor Process Capabilities Are

Effective

• SEI has developed criteria for determining organizations’

capability to develop software effectively.

• SEI’s criteria defines five levels of development

capability, ranging from level 1 (ad hoc and chaotic) to

level 5 (optimized).

• Lockheed Martin-Mission Systems has been

independently evaluated as an SEI Level 5 development

organization, and DCS 2000 was one of three systems

included in the evaluation determination.
Appendix I

Briefing on the Decennial Census

Page 85 GAO/AIMD-00-61 2000 Census Data Capture System

68

Audit Results: DCS 2000 Risk Mitigators

4. Development Contractor Process Capabilities Are

Effective (cont.)

• Among the effective software development key processes

that Lockheed Martin-Mission Systems has are

configuration management; project planning, tracking,

and oversight; and quality assurance.
Appendix I

Briefing on the Decennial Census

Page 86 GAO/AIMD-00-61 2000 Census Data Capture System

69

Conclusions

The Census Bureau has made reasonable progress in acquiring and

deploying DCS 2000, and under normal circumstances, where completion

deadlines can be changed and some degree of schedule slippage is

tolerable, delivering promised DCS 2000 capabilities would not represent a

significant risk. However, the demands of the decennial census necessitate

that DCS 2000 be developed, tested, deployed, and ready to operate in just

3 months, with no allowance for slippage.

Given DCS 2000’s current status and quality, completing DCS 2000 by

March 6, 2000, poses considerable risk. In particular, significant system

functionality remains to be developed, acceptance tested (e.g., 24 percent of

the total number of DCS 2000 high-level functional requirements have not yet

been acceptance tested), deployed, and site tested. Further, a key measure

of system quality (trends in unfixed severe defects) is not yet moving in a

direction consistent with that of a maturing system. Given that (1) a number

of significant test events have yet to occur and (2) these yet-to-occur events

are likely to uncover more defects, the chances of the defect trend moving in

a favorable direction soon are uncertain. Further, given that the average
Appendix I

Briefing on the Decennial Census

Page 87 GAO/AIMD-00-61 2000 Census Data Capture System

70

Conclusions (cont.)

time to fix severe DCS 2000 defects over the last 6 months (about 10 days)

would consume the time between conclusion of the later test events and the

day that DCS 2000 must be operational, the chances of all specified system

capabilities being ready on time are reduced.

To meet the challenge of a very demanding and fault-intolerant schedule, the

bureau has been following a plan that provides for (1) development and

testing of DCS 2000 in a series of increments, called releases, and

(2) deployment and site testing of the releases as they become ready. This

concurrent development and testing approach, while introducing risk due to

the concurrent activities, allows early system capability to get into the hands

of the end users faster, which could save time. Further, while this concurrent

approach does increase the risk of contention for test resources and

uncontrolled changes to the system baseline configuration (because of the

number of development and test baselines being used), these risks can be

mitigated by effective system development capabilities, such as project risk

management, configuration management, project planning, and oversight. In

DCS 2000’s case, these activities are being performed by the development
Appendix I

Briefing on the Decennial Census

Page 88 GAO/AIMD-00-61 2000 Census Data Capture System

71

Conclusions (cont.)

contractor, which has been independently certified as an SEI level 5 (on a 5

level scale) software developer. Such contractor capability is an important

risk mitigator, not only with respect to each of these activities, but for all

remaining DCS 2000 development, testing, and deployment activities.

Nevertheless, the schedule challenge facing the bureau on DCS 2000,

particularly in light of where the system currently stands and what remains to

be accomplished, are formidable. Both bureau and development contractor

officials agree. Accordingly, they have initiated proactive and appropriate

risk management steps. While these steps do not guarantee success, they

should minimize the probability of the risk occurring and the impact on the

program if it does.
Appendix I

Briefing on the Decennial Census

Page 89 GAO/AIMD-00-61 2000 Census Data Capture System

72

Appendixes

• Appendix I: Scope and Methodology

• Appendix II: Description of Other Decennial Census

Systems
Appendix I

Briefing on the Decennial Census

Page 90 GAO/AIMD-00-61 2000 Census Data Capture System

73

• We determined DCS 2000 status and quality by:

– Reviewing system development plans, project

management plans, activity plans (test, deployment,

etc.), project status reports, program management

briefings, system functional requirements and

specifications, and test results reports.

– Comparing original to revised plans and current plans

to status reports.

– Observing DCS 2000 OTDR in Baltimore.

– Attending periodic DCS 2000 project review meetings

between the bureau and the development contractor.

Appendix I

Scope and Methodology
Appendix I

Briefing on the Decennial Census

Page 91 GAO/AIMD-00-61 2000 Census Data Capture System

74

– Interviewing bureau DCS 2000 project management

and contractor officials.

– Identifying how the bureau measured maturity and

quality of DCS 2000 throughout its development life

cycle, and analyzing these and other measures for

indicators of progress and system health (e.g., PTR

trends and age and test results).

– Reviewing the scope and results of DCS 2000 testing

activities, including test plans and results.

Appendix I

Scope and Methodology (cont.)
Appendix I

Briefing on the Decennial Census

Page 92 GAO/AIMD-00-61 2000 Census Data Capture System

75

• We determined DCS 2000 risks and risk mitigators by:

– Determining bureau and development contractor key

management processes (e.g., requirements

management, configuration management, project

planning, and project tracking and oversight).

– Analyzing the results of status and quality

determinations vis-à-vis DCS 2000 expectations and

remaining steps to complete.

– Analyzing plans for completing DCS 2000 vis-à-vis

generally accepted system engineering principles.

– Interviewing bureau DCS 2000 project managers and

contractor officials.

Appendix I

Scope and Methodology (cont.)
Appendix I

Briefing on the Decennial Census

Page 93 GAO/AIMD-00-61 2000 Census Data Capture System

76

• We briefed bureau and development contractor officials

on the results of our work and incorporated updated

information they provided, as appropriate.

• We performed our work at the bureau’s headquarters in

Suitland, MD; the DCS 2000 program office in Lanham,

MD; bureau and development contractor facilities in

Bowie, MD; and the bureau’s DCC in Baltimore, MD from

June 1999 through December 1999 in accordance with

generally accepted government auditing standards.

Appendix I

Scope and Methodology (cont.)
Appendix I

Briefing on the Decennial Census

Page 94 GAO/AIMD-00-61 2000 Census Data Capture System

77

Appendix II

Description of Decennial Census Systems

• Geographic Support System provides the basic census

address list, maps, and geographic reference files for all

census programs, including the 2000 decennial census.

• Pre-Appointment Management System/Automated

Decennial Administrative Management System is the

administrative management system that supports

temporary bureau employee applicant tracking and

processing, temporary employee selection records,

recruiting reports, personnel and payroll processing, and

archiving of historical employment data.
Appendix I

Briefing on the Decennial Census

Page 95 GAO/AIMD-00-61 2000 Census Data Capture System

78

Appendix II

Description of Decennial Census Systems (cont.)

• Operations Control System 2000 supports, manages, and

controls all field operations for Census 2000.

• Management Information System 2000 is the official

source of management information on Census 2000 and

will provide information on scheduling, progress to date,

cost to date compared to budget, and performance

anomalies.
Appendix I

Briefing on the Decennial Census

Page 96 GAO/AIMD-00-61 2000 Census Data Capture System

79

Appendix II

Description of Decennial Census Systems (cont.)

• Telephone Questionnaire Assistance and Coverage

Edit Follow-Up

– TQA is a telephone response system that uses

both automated interactive voice response

technology and live operator responses to ensure

that the public’s needs are addressed.

– CEFU flags edit failures on census forms and

sends them to a telephone follow-up operation

where a generic follow-up interview will be

conducted.
Appendix I

Briefing on the Decennial Census

Page 97 GAO/AIMD-00-61 2000 Census Data Capture System

80

Appendix II

Description of Decennial Census Systems (cont.)

• Internet Data Collection/Internet Questionnaire Assistance

– IDC provides respondents, on a limited basis, the ability

to complete their short form questionnaire on-line over

the Internet.

– IQA provides respondents with answers to questions via

the Internet.

• Accuracy and Coverage Evaluation is a follow-up survey

for a representative sample of 300,000 housing units

across the nation.
Appendix I

Briefing on the Decennial Census

Page 98 GAO/AIMD-00-61 2000 Census Data Capture System

81

Appendix II

Description of Decennial Census Systems (cont.)

• Headquarters processing is a group of systems,

applications, and processes performed on the data

collected by various Census 2000 systems during the

census and during the ACE phases. Headquarters

processing is also responsible for creating final census

data files.

• Data Access and Dissemination System consists of two

primary subsystems, American FactFinder and Data

Products Production. These subsystems jointly provide

access to census results through the Internet.
Page 99 GAO/AIMD-00-61 2000 Census Data Capture System

Appendix II

Comments From the Department of 

Commerce ApendxiI

Note: GAO comments 

supplementing those in the 

report text appear at the 

end of this appendix.
Appendix II

Comments From the Department of 

Commerce

Page 100 GAO/AIMD-00-61 2000 Census Data Capture System

See comment 1.

See comment 2.
Appendix II

Comments From the Department of 

Commerce

Page 101 GAO/AIMD-00-61 2000 Census Data Capture System

The following are GAO’s comments on the Department of Commerce’s 

January 27, 2000, letter responding to a draft of this report.

GAO Comments 1. It was not our intent to imply this. As we discuss throughout the report, 

DCS 2000 development, testing, and deployment has been ongoing on 

an incremental basis and we clearly identify what activities have been 

accomplished to date. Further, we identify what development, test, and 

deployment activities remain to be accomplished, and we characterize 

these remaining activities as significant and formidable in light of the 

short time remaining before DCS 2000 must be operational. 

Nevertheless, we have modified the sentence that the bureau cited to 

eliminate the possibility of misinterpretation.

2. We agree that the 16-day average time for resolving all PTRs is not a 

good indicator of the time necessary to resolve emergency PTRs 

because all PTRs include five different categories of PTRs and each 

category receives a different priority, emergency being the most severe 

category and the highest priority for resolution. For this reason, we do 

not state anywhere in our report that the 16-day average time needed 

over the last 6 months (which covered June through November 1999) to 

resolve all PTRs is the operative indicator of the time to resolve the 

most severe category of PTRs (emergency). Rather, the operative 

indicator that we cite in the report for the time necessary to resolve 

severe PTRs is 10 days, which is the average time that the bureau’s own 

data show it has taken over the last 6 months to resolve both 

emergency and critical PTRs. We focused on these two most severe 

categories of PTRs in our analysis because the bureau’s own definition 

of emergency and critical PTRs (emergency PTRs are problems that 

prevent system operation and critical PTRs are problems that 

significantly degrade system operation) mean that both of these 

categories of PTRs will have to be resolved before March 6, 2000, if 

DCS 2000 is to operate as intended. Accordingly, we state in the report 

that “it has been taking an average of 16 days to resolve all PTRs over 

the last 6 months and 10 days for severe PTRs.”

(511178) Letr
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the 

following address, accompanied by a check or money order made 

out to the Superintendent of Documents, when necessary, VISA and 

MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and 

testimony. To receive facsimile copies of the daily list or any list 

from the past 30 days, please call (202) 512-6000 using a touchtone 

phone. A recorded menu will provide information on how to obtain 

these lists.

For information on how to access GAO reports on the INTERNET, 

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at: 

http://www.gao.gov

United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00