Report to Congressional Requesters

United States Government Accountability Office

GAO 

INFORMATION 

TECHNOLOGY 

Census Bureau Testing 

of 2010 Decennial 

Systems Can Be 

Strengthened 

March 2009 

GAO-09-262 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

March 2009

INFORMATION TECHNOLOGY 

Census Bureau Testing of 2010 Decennial Systems

Can Be Strengthened 

Highlights of GAO-09-262, a report to 

congressional requesters 

The Decennial Census is mandated 

by the U.S. Constitution and 

provides vital data that are used, 

among other things, to reapportion

and redistrict congressional seats.

In March 2008, GAO designated the 

2010 Decennial Census a high-risk 

area, citing a number of longstanding and emerging challenges,

including weaknesses in the 

Census Bureau’s (Bureau) 

management of its information 

technology (IT) systems and 

operations. In conducting the 2010 

census, the Bureau is relying on 

both the acquisition of new IT 

systems and the enhancement of

existing systems. Thoroughly 

testing these systems before their 

actual use is critical to the success

of the census. GAO was asked to

determine the status of and plans 

for testing key decennial systems.

To do this, GAO analyzed testing 

documentation, interviewed 

Bureau officials and contractors,

and compared the Bureau’s efforts 

with recognized best practices.

What GAO Recommends

GAO is recommending that the 

Secretary of Commerce direct the 

Bureau to complete key system

testing activities, develop and 

maintain plans for integration 

testing, and improve the oversight

of and guidance for systems 

testing. In comments on a draft of 

this report, the department agreed

with GAO’s recommendations. 

Although the Bureau has made progress in testing key decennial systems, 

critical testing activities remain to be performed before systems will be ready 

to support the 2010 census. Bureau program offices have completed some 

testing of individual systems, but significant work still remains to be done, and 

many plans have not yet been developed (see table below). In its testing of 

system integration, the Bureau has not completed critical activities; it also 

lacks a master list of interfaces between systems; has not set priorities for the 

testing of interfaces based on criticality; and has not developed testing plans 

and schedules. Although the Bureau had originally planned what it refers to as 

a Dress Rehearsal, starting in 2006, to serve as a comprehensive end-to-end 

test of key operations and systems, significant problems were identified 

during testing. As a result, several key operations were removed from the 

Dress Rehearsal and did not undergo end-to-end testing. The Bureau has 

neither developed testing plans for these key operations, nor has it 

determined when such plans will be completed. 

Weaknesses in the Bureau’s testing progress and plans can be attributed, in 

part, to a lack of sufficient executive-level oversight and guidance. Bureau 

management does provide oversight of system testing activities, but the 

oversight activities are not sufficient. For example, Bureau reports do not 

provide comprehensive status information on progress in testing key systems 

and interfaces, and assessments of the overall status of testing for key 

operations are not based on quantitative metrics. Specifically, key operations 

that do not yet have plans developed are marked as making acceptable 

progress based solely on management judgment. Further, although the Bureau 

has issued general testing guidance, it is neither mandatory nor specific 

enough to ensure consistency in conducting system testing. Without adequate 

oversight and more comprehensive guidance, the Bureau cannot ensure that it 

is thoroughly testing its systems and properly prioritizing testing activities 

before the 2010 Decennial Census, posing the risk that these systems may not 

perform as planned. 

Status and Plans of 2010 System Testing 

System Testing status

Testing plan 

completed 

Testing schedule

completed 

Headquarters processing In progress Partial Partial 

Master address and geographic 

information In progress Partial Partial

Decennial response integration In progress Partial Partial 

Field data collection automation In progress Partial Partial 

Paper-based operations In progress No Partial 

Data access and dissemination In progress Partial Partial 

Source: GAO analysis of Bureau data. 

To view the full product, including the scope 

and methodology, click on GAO-09-262. 

For more information, contact David Powner 

at (202) 512-9286 or pownerd@gao.gov. 
Contents 

Letter 1

Background 2

Bureau Is Making Progress in Conducting Key Decennial System 

Testing, but Lacks Plans and Schedules to Guide Remaining 

Efforts 11

Conclusions 26

Recommendations for Executive Action 26 

Agency Comments and Our Evaluation 27 

Appendix I Scope and Methodology 30 

Appendix II Comments from the Department of Commerce 32 

Appendix III GAO Contact and Staff Acknowledgments 34 

Tables 

Table 1: Key Systems Supporting 2010 Census 5 

Table 2: Status of System Testing and Plans 11 

Table 3: Status of System Testing and Plans for Headquarters 

Processing Systems (Dress Rehearsal and 2010 Testing) 12 

Table 4: Status of System Testing and Plans for MAF/TIGER (2010

Testing) 14

Table 5: Status of System Testing and Plans for DRIS 

(2010 Testing) 15 

Table 6: Status of System Testing and Plans for FDCA 

(2010 Testing) 16 

Table 7: Status of System Testing and Plans for PBO Release 0 

(2010 Testing) 18 

Table 8: Status of System Testing and Plans for DADS II 

Components (2010 Testing) 19 

Table 9: Integration Testing Status 20 

Page i GAO-09-262 Census Testing
Figures 

Figure 1: Summary of Key Decennial Activities 3 

Figure 2: Timeline of Key Decennial Activities 4 

Figure 3: Dates Key Decennial Systems Are to Be Operational 6 

Figure 4: Inventory of Testing Activities as of May 2008 24 

Abbreviations 

Bureau Census Bureau 

DADS II Data Access and Dissemination System II 

DRIS Decennial Response Integration System 

FDCA Field Data Collection Automation 

IEEE Institute of Electrical and Electronics Engineers 

IT information technology 

MAF/TIGER Master Address File/Topologically Integrated Geographic 

Encoding and Referencing system 

PBO Paper-Based Operations 

RDS Replacement Dissemination System 

RPS Response Processing System 

RTS Replacement Tabulation System 

UC&M Universe Control and Management 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page ii GAO-09-262 Census Testing
United States Government Accountability Office

Washington, DC 20548 

 

March 5, 2009

Congressional Requesters

The Census Bureau (Bureau) is relying on both the acquisition of new 

systems and the enhancement of existing legacy systems for conducting 

operations for the 2010 Decennial Census. As you know, the census is 

mandated by the U.S. Constitution and provides data that are vital to the 

nation. These data are used, for example, to reapportion and redistrict the 

seats of the U.S. House of Representatives; realign the boundaries of the 

legislative districts of each state; allocate billions of dollars in federal 

financial assistance; and provide a social, demographic, and economic 

profile of the nation’s people to guide policy decisions at each level of 

government. The Bureau is required to take a population count as of April 

1, 2010 (Census Day), and the Secretary of Commerce is required to report 

to the President on the tabulation of total population by state within nine 

months of that date.1

Carrying out the census is the responsibility of the Department of 

Commerce’s Census Bureau, which is relying on automation and 

technology to improve the coverage, accuracy, and efficiency of the 2010 

census. Because the accuracy of the 2010 census depends, in part, on the 

proper functioning of these systems, both individually and when 

integrated, thorough testing of these systems before their actual use is 

critical to the success of the census. 

In March 2008, we designated the 2010 Decennial Census as a high-risk 

area, citing a number of long-standing and emerging challenges,2

 including 

weaknesses in the Bureau’s management of its information technology 

(IT) systems and operations. The 2010 Decennial Census remained as one

of our high-risk areas in our recent high-risk update issued in January 

2009.

3

 Given the importance of comprehensive testing prior to the 2010 

census, you asked us to determine the status of and plans for the testing of 

key decennial systems. 

 1

13 U.S.C. 141 (a) and (b). 

2

GAO, Information Technology: Significant Problems of Critical Automation Program

Contribute to Risks Facing 2010 Census, GAO-08-550T (Washington, D.C.: Mar. 5, 2008).

3

GAO, High-Risk Series: An Update, GAO-09-271 (Washington, D.C.: Jan. 22, 2009). 

Page 1 GAO-09-262 Census Testing
 

To address this objective, we analyzed documentation related to system, 

integration, and end-to-end testing,4

 including plans, schedules, and 

results, and interviewed Bureau officials and contractors. We then 

compared the Bureau’s practices with those identified in our testing guide 

and other best practices.5 We conducted this performance audit from June 

2008 to February 2009, in accordance with generally accepted government 

auditing standards. Those standards require that we plan and perform the 

audit to obtain sufficient, appropriate evidence to provide a reasonable 

basis for our findings and conclusions based on our audit objective. We 

believe that the evidence obtained provides a reasonable basis for our 

findings and conclusions based on our audit objective. Appendix I 

contains further details about our scope and methodology. 

The Bureau’s mission is to provide comprehensive data about the nation’s

people and economy. Its core activities include conducting decennial, 

economic, and government censuses; conducting demographic and 

economic surveys; managing international demographic and 

socioeconomic databases; providing technical advisory services to foreign 

governments; and performing other activities such as producing official

population estimates and projections.

Background 

Conducting the decennial census is a major undertaking that includes the 

following major activities: 

• Establishing where to count. This includes identifying and correcting 

addresses for all known living quarters in the United States (address 

canvassing) and validating addresses identified as potential group 

quarters, such as college residence halls and group homes (group quarters 

validation). 

• Collecting and integrating respondent information. This includes 

delivering questionnaires to housing units by mail and other methods,6

 4

System testing verifies that a system meets specified requirements. Integration testing 

verifies that systems, when combined, work as intended. End-to-end testing verifies that a 

set of systems work as intended in an operational environment. 

5

See, for example, GAO, Year 2000 Computing Crisis: A Testing Guide, 

GAO/AIMD-10.1.21 (Washington, D.C.: Nov. 1, 1998); and IEEE Std 12207-2008, Systems 

and Software Engineering-Software Life Cycle Processes (Piscataway, N.J.: 2008). 

6

For example, in the “update/leave” operation, after enumerators update addresses, they 

leave questionnaires at housing units; this occurs mainly in rural areas that lack street 

names, house numbers, or both. 

Page 2 GAO-09-262 Census Testing
 

processing the returned questionnaires, and following up with 

nonrespondents through personal interviews (nonresponse follow-up). It 

also includes enumerating residents of group quarters (group quarters 

enumeration) and occupied transitional living quarters (enumeration of 

transitory locations), such as recreational vehicle parks, campgrounds, 

and hotels. It also includes a final check of housing unit status (field 

verification) where Bureau workers verify potential duplicate housing 

units identified during response processing. 

• Providing census results. This includes processes to tabulate and 

summarize census data and disseminate the results to the public. 

Figure 1 illustrates key decennial activities. 

Figure 1: Summary of Key Decennial Activities 

Source: GAO analysis of Bureau data. 

Establish where to count Collect and integrate respondent information Provide census results

Lists of addresses for 

living quarters in the 

United States are 

compiled from multiple 

sources, such as the 

Postal Service and state 

and local governments.

The addresses and other 

information are verified by 

enumerators using handheld 

computers (address canvassing). 

Group quarters, such as college 

residence halls and group 

homes, are validated using paper 

(group quarters validation).

A final check of 

housing unit status, 

known as field 

verification, is

conducted by 

census workers

to verify potential 

duplicate housing 

units (field 

verification).

The data are 

used to 

establish a

universe of 

addresses from 

which to count.

Questionnaires are 

delivered to housing units

by mail and other methods, 

such as enumerators

(update/leave).

Enumerators contact 

nonrespondents

through personal 

interviews to complete 

paper questionnaires.

Returned 

questionnaires

are processed.

Enumerators also collect information 

from other living quarters, such as

group quarters (group quarters

enumeration) and occupied transitory 

locations, such as hotels (enumeration 

of transitory locations), using paper.

Census data are tabulated, 

summarized, and 

disseminated to the public.

Apportionment data used to 

redistribute seats in the 

House of Representatives

are delivered to the 

President, and redistricting 

data are delivered to state 

legislatures.

Census

The 2010 census enumerates the number and location of people on Census 

Day, which is April 1, 2010. However, census operations begin long before 

Census Day and continue afterward. For example, address canvassing for the 

2010 census will begin in April 2009, while tabulated census data must be 

distributed to the President by December 31, 2010, and to state legislatures by

March 31, 2011. Figure 2 presents a timeline of key decennial operations. 

Page 3 GAO-09-262 Census Testing
 

Figure 2: Timeline of Key Decennial Activities 

2008 2009

J FMAM J J A S OND J FMAM J J A S OND

2010

J FMAM J J A S OND

2011

J FMAM J J A S OND

Source: GAO summary of Bureau data. 

Opening of 12 

regional census

centers

Opening of 151 

census offices

Census Day

Deliver 

apportionment 

counts to the 

President

Start of address

canvassing 

operations

Start of 

group 

quarters

validation

Initial questionnaires

mailed

Begin nonresponse 

follow-up operations

Deliver 

redistricting 

data to state 

legislatures

Opening of 344 

census offices

(495 total)

Role of IT in the Decennial 

Census 

Automation and IT are to play a critical role in the success of the 2010 

census by supporting data collection, analysis, and dissemination. Several 

systems will be used in the 2010 census. For example, enumeration 

“universes,” which serve as the basis for enumeration operations and 

response data collection, are organized by the Universe Control and 

Management (UC&M) system, and response data are received and edited 

to help eliminate duplicate responses using the Response Processing 

System (RPS). Both UC&M and RPS are legacy systems that are 

collectively called the Headquarters Processing Systems. 

Geographic information and support to aid the Bureau in establishing where 

to count U.S. citizens are provided by the Master Address File/Topologically 

Integrated Geographic Encoding and Referencing (MAF/TIGER) system. The

Decennial Response Integration System (DRIS) is to provide a system for

collecting and integrating census responses from all sources, including forms

and telephone interviews. The Field Data Collection Automation (FDCA) 

program includes the development of handheld computers for the address 

canvassing operation and the systems, equipment, and infrastructure that

field staff will use to collect the data. Paper-Based Operations (PBO) was

established in August 2008, primarily to handle some of the operations that

were originally part of FDCA. PBO includes IT systems and infrastructure

needed to support the use of paper forms for operations such as group

quarters enumeration activities, nonresponse follow-up activities,

enumeration at transitory locations activities, and field verification activities.

These activities were originally to be conducted using IT systems and 

infrastructure developed by the FDCA program. Finally, the Data Access and

Page 4 GAO-09-262 Census Testing
 

Dissemination System II (DADS II) is to replace legacy systems for tabulating

and publicly disseminating data. 

Table 1 describes the key systems supporting the 2010 census, as well as 

the offices responsible for their development. 

Table 1: Key Systems Supporting 2010 Census 

System name Responsible entity Description 

Headquarters Processing—

Universe Control and 

Management (UC&M) 

Decennial System 

and Processing

Office 

Organizes address files into enumeration “universes,” which serve as the basis 

for enumeration operations and response data collection. UC&M data contain, 

among other things, a list of addresses to which census respondent forms must 

be delivered. This functionality is critical to ensuring that census respondent 

forms are delivered to the correct addresses. 

Headquarters Processing—

Response Processing 

System (RPS) 

Decennial System 

and Processing

Office 

Receives response data and edits the data to help eliminate duplicate 

responses by, for example, identifying people who have been enumerated more 

than once. After response data are finalized, they are provided to DADS II for 

tabulation and dissemination. 

Master Address 

File/Topologically Integrated

Geographic Encoding and 

Referencing (MAF/TIGER) 

system 

Geography Division Provides geographic information and support to aid the Bureau in establishing 

where to count the U.S. population for the 2010 census. The Bureau’s address 

list—MAF—is associated with the TIGER system, which is a geographic 

information system containing street maps and other geographic features. 

MAF/TIGER was recently updated under the MAF/TIGER Accuracy 

Improvement Project in April 2008, which provided corrected coordinates on a 

county-by-county basis for all current features in the TIGER database. 

Decennial Response 

Integration System (DRIS) 

DRIS Program 

Management Office 

and Lockheed Martin 

(lead contractor) 

Collects and integrates census responses from all sources, including forms and 

telephone interviews. DRIS is to improve accuracy and timeliness by 

standardizing the response data and providing the data to other Bureau 

systems for analysis and processing. 

Field Data Collection 

Automation (FDCA) 

FDCA Program

Management Office 

and Harris (lead 

contractor) 

Provides automation support for field data collection operations. It includes the 

development of handheld computers for the address canvassing operation and 

the systems, equipment, and infrastructure that field staff will use to collect

data. It is to establish office automation for the 12 regional census centers, the 

Puerto Rico area office, and approximately 494 temporary local census offices. 

FDCA handheld computers were originally to be used for nonresponse 

followup, but due to problems in testing, nonresponse followup was switched to 

paper-based operations. 

Paper-Based Operations 

(PBO) 

Decennial System 

and Processing

Office 

Established in August 2008, primarily to handle operations that were originally 

part of FDCA. PBO includes IT systems and infrastructure needed to support 

the use of paper forms for operations such as group quarters enumeration,

nonresponse followup, enumeration at transitory locations, and field verification.

Data Access and 

Dissemination System II 

(DADS II) 

DADS II Program 

Management Office 

and IBM (lead 

contractor) 

Replaces legacy systems for tabulating and publicly disseminating data. DADS 

II includes the Replacement Tabulation System and the Replacement 

Dissemination System, which are expected to maximize the efficiency, 

timeliness, and accuracy of tabulation and dissemination products and services; 

minimize the cost of tabulation and dissemination; and increase user 

satisfaction with related services. The Replacement Tabulation System is 

responsible for tabulating 2010 census data. The Replacement Dissemination 

System is responsible for distributing and disseminating census results. 

Source: GAO analysis of Bureau data. 

Page 5 GAO-09-262 Census Testing
 

Figure 3 shows the timeframes when each of the systems for the 2010 

census are to be operational, according to the Census Bureau. 

Figure 3: Dates Key Decennial Systems Are to Be Operational 

2009

J FMAMJ J A S OND

2008 2010

J FMAMJ J A S OND

2007

J FMAMJ J A S OND J FMAMJ J A S OND

2011

J FMAMJ J A S OND

Source: GAO summary of Bureau data. 

MAF/TIGER

UC&M

RPS

DRIS

FDCA

PBO

DADS II

We Have Previously

Reported on Weaknesses 

in Management of Census 

IT Systems 

We have reported long-standing weaknesses in the Bureau’s management 

of its IT systems. For example, in October 2007, we reported on the status

and plans of key 2010 census IT acquisitions and whether the Bureau was 

adequately managing associated risks.7

 We identified critical weaknesses 

in the Bureau’s risk management practices, including those associated 

with risk identification, mitigation, and executive-level oversight. Further,

operational testing planned during the census Dress Rehearsal would take

place without the full complement of systems and functionality that was 

originally planned, the Bureau had not finalized its plans for testing all the 

systems, and it was unclear whether the plans would include testing to 

address all interrelated systems and functionality. We recommended that

the Bureau develop a comprehensive plan to conduct an end-to-end test of 

its systems under census-like conditions. 

 7

GAO, Information Technology: Census Bureau Needs to Improve Its Risk Management 

of Decennial Systems, GAO-08-79 (Washington, D.C.: Oct. 5, 2007). 

Page 6 GAO-09-262 Census Testing
 

In March 2008, we designated the 2010 census as a high-risk area, citing 

several long-standing and emerging challenges.8

 These challenges included, 

among other things, weaknesses in risk management and system testing,

elimination of several operations from the 2008 Dress Rehearsal, and 

questions surrounding the performance of handheld computers developed for 

the 2010 census. We have also testified9

 on significant risks facing the 2010

census. For example, in March 2008, we testified that the FDCA program was

experiencing significant problems, including schedule delays and cost

increases resulting from changes to system requirements, which required

additional work and staffing. Shortly thereafter, in April 2008, we testified on

the Bureau’s efforts to implement risk reduction strategies, including the 

decision to drop the use of handheld computers during the nonresponse 

follow-up operation and revert to a paper-based operation. Further, in June

2008, we testified that the Bureau had taken important steps to plan for a 

paper-based nonresponse follow-up operation, but several aspects remained

uncertain. We concluded that it was critical to test capabilities for supporting 

the nonresponse follow-up operation.

In July 2008, we reported that continued planning and testing of the 

handheld computers would be critical to the address canvassing 

operation.10 Specifically, the Bureau had developed a testing plan that 

included a limited operational field test, but the plan did not specify the 

basis for determining whether the FDCA solution was ready for address 

canvassing and when and how this determination would occur. 

In response to our findings and recommendations, the Bureau has taken 

several steps to improve its management of the 2010 Decennial Census. 

For example, the Bureau has sought external assessments of its activities 

from independent research organizations, implemented a new 

management structure and management processes, brought in 

experienced personnel in key positions, and established improved 

reporting processes and metrics. 

 8

GAO-08-550T; GAO, Census 2010: Census at Critical Juncture for Implementing Risk 

Reduction Strategies, GAO-08-659T (Washington, D.C.: Apr. 9, 2008). 

9

GAO-08-550T; GAO, 2010 Census: Census at Critical Juncture for Implementing Risk 

Reduction Strategies, GAO-08-685T (Washington, D.C.: Apr. 15, 2008) and GAO, 2010 

Census: Plans for Decennial Census Operations and Technology Have Progressed, But 

Much Uncertainty Remains, GAO-08-886T (Washington, D.C.: June 11, 2008). 

10GAO, 2010 Census: Census Bureau’s Decision to Continue with Handheld Computers 

for Address Canvassing Makes Planning and Testing Critical, GAO-08-936 (Washington,

D.C.: July 31, 2008). 

Page 7 GAO-09-262 Census Testing
 

As stated in our testing guide and the Institute of Electrical and 

Electronics Engineers (IEEE) standards,11 complete and thorough testing 

is essential for providing reasonable assurance that new or modified IT 

systems will perform as intended. To be effective, testing should be 

planned and conducted in a structured and disciplined fashion that 

includes processes to control each incremental level of testing, including

testing of individual systems, the integration of those systems, and testing 

to address all interrelated systems and functionality in an operational 

environment. 

• System testing: verifies that the complete system (i.e., the full complement 

of application software running on the target hardware and systems 

software infrastructure) meets specified requirements. It allows for the 

identification and correction of potential problems within an individual

system, prior to integration with other systems. 

• Integration testing: verifies that systems, when combined, work together

as intended. Effective integration testing ensures that external interfaces 

work correctly and that the integrated systems meet specified

requirements. 

• End-to-end testing: verifies that a defined set of interrelated systems, 

which collectively support an organization’s core business area or 

function, interoperate as intended in an operational environment. The 

interrelated systems include not only those owned and managed by the 

organization, but also the external systems with which they interface. 

To be effective, this testing should be planned and scheduled in a 

structured and disciplined fashion. Comprehensive testing that is 

effectively planned and scheduled can provide the basis for identifying key 

tasks and requirements and better ensure that a system meets these 

specified requirements and functions as intended in an operational 

environment. 

In preparation for the 2010 census, the Bureau planned what it refers to as 

the Dress Rehearsal. The Dress Rehearsal is managed by the Bureau’s 

Decennial Management Division, in collaboration with other Bureau 

divisions (including the program offices, shown in table 1, which are 

responsible for developing and testing each of the systems). The Dress 

Comprehensive Testing 

Improves Chances of a 

Successful Decennial 

Census 

Dress Rehearsal Includes 

Testing of Certain Systems 

and Operations 

 11GAO/AIMD-10.1.21 and IEEE Std. 12207-2008. 

Page 8 GAO-09-262 Census Testing
 

Rehearsal includes systems and integration testing,12 as well as end-to-end 

testing of key operations in a census-like environment. During the Dress 

Rehearsal period, running from February 2006 through June 2009, the 

Bureau is developing and testing systems and operations, and it held a 

mock Census Day on May 1, 2008. The Dress Rehearsal activities, which 

are still under way, are a subset of the activities planned for the actual 

2010 census and include testing of both IT and non-IT related functions, 

such as opening offices and hiring staff. 

The Dress Rehearsal tested several activities involving key systems. For 

example, the Bureau tested key systems with address canvassing and 

group quarters validation operations, including FDCA handheld computers 

and the MAF/TIGER system. In addition, the Bureau used the UC&M 

system and MAF/TIGER to provide an initial list of housing unit addresses 

for the Dress Rehearsal test sites. Questionnaires were mailed to these 

housing units in April 2008. Subsequently, a mock Census Day was held on

May 1, 2008—1 month later than originally planned. The mock Census Day 

was delayed, in part, to focus greater attention on testing the technology 

being used. 

The Dress Rehearsal identified significant technical problems during the 

address canvassing operations. For example, the Bureau had originally 

planned to use handheld computers, developed under the FDCA program, 

for operations such as address canvassing and non-response followup. 

However, from May 2007 to June 2007, the Bureau tested the handhelds 

under census-like conditions for the first time during the Dress Rehearsal 

address canvassing operation. Bureau officials observed a number of 

performance problems with the handheld computers, such as slow and 

inconsistent data transmissions.13 In addition, help desk logs revealed that 

users had frequently reported problems, such as the devices freezing up or

users having difficulties collecting mapping coordinates and working with 

large blocks (geographic areas with large numbers of housing units, more 

often found in urban areas). 

The Bureau also found system problems during testing of the group 

quarters validation operation, in which field staff validate addresses as 

Dress Rehearsal Testing of Key 

Systems and Activities 

Identified Problems with 

Technologies 

 12Individual program offices manage individual system testing for the Dress Rehearsal, and 

integration testing is managed by the pairs of program offices whose interfaces are being

tested. 

13For more information on performance of the handheld computers, see GAO-08-936. 

Page 9 GAO-09-262 Census Testing
 

group quarters and collect information required for their later 

enumeration. As part of this operation, the Bureau tested the operations 

control system—designed to manage field operations that rely on paper, as 

well as those that rely on the handheld computers—and the system was 

found to be unreliable. As a result, the workload for these operations had 

to be supplemented with additional paper-based efforts by local census 

office staff, instead of being performed electronically, as intended. 

As a result of the problems observed with the handheld computers and 

operations control system, cost overruns and schedule slippage in the 

FDCA program, and other issues, the Bureau removed the planned testing 

of key operations from the Dress Rehearsal as follows: 

Results of Dress Rehearsal 

Testing Led to Decisions to 

Remove Testing of Certain 

Operations and to Revert to 

Some Paper-Based Processes 

for Key Operations • update/leave (that is, after enumerators update addresses, they leave 

questionnaires at housing units; this occurs mainly in rural areas lacking 

street names, house numbers, or both), 

• nonresponse follow-up, 

• enumeration of transitory locations,

• group quarters enumeration, and 

• field verification. 

Furthermore, in April 2008, the Secretary of Commerce announced a 

redesign of the 2010 Decennial Census, including the FDCA program. 

Specifically, the Bureau would no longer use handheld computers for 

nonresponse follow-up (its largest field operation), but would conduct 

paper-based nonresponse follow-up, as in previous censuses. It would, 

however, continue to use the handheld computers for the address 

canvassing operations. In May 2008, the Bureau issued a plan that detailed

key components of the paper-based operation and described processes for 

managing it and other operations. It later established the PBO office to 

manage designing, developing, and testing paper-based operations, as well 

as to prepare related training materials. 

In addition to the planned Dress Rehearsal testing, the Bureau is planning 

supplementary testing to prepare for the 2010 Decennial Census. This 

testing includes system, integration, and end-to-end testing of changes 

resulting from the Dress Rehearsal, operations or features that were not 

tested during the Dress Rehearsal, and additional features or 

enhancements that are to be added after the Dress Rehearsal. 

Additional Testing Is Planned 

to Supplement the Dress 

Rehearsal 

Page 10 GAO-09-262 Census Testing
 

The Bureau has made progress in conducting system, integration, and endto-end testing for the 2010 census, but much remains to be done. 

Significant testing remains to be done, and many plans for the remaining 

testing activities have not been developed. The weaknesses in the Bureau’s

IT testing can be attributed, in part, to a lack of sufficient executive-level 

oversight and guidance on testing. Without comprehensive oversight and 

guidance, the Bureau cannot ensure that it is thoroughly testing its 

systems before the 2010 Decennial Census. 

Through the Dress Rehearsal and other testing activities, the Bureau has

completed key system tests, but significant testing has yet to be performed,

and planning for this is not complete. For example, the Headquarters 

Processing systems (UC&M and RPS) are still completing system testing

related to the Dress Rehearsal, and the program office is planning for further

testing. For DRIS, on the other hand, system testing related to the Dress 

Rehearsal is complete, and additional 2010 system testing is under way. Table 

2 summarizes the status and plans for system testing. 

Bureau Is Making 

Progress in 

Conducting Key 

Decennial System 

Testing, but Lacks 

Plans and Schedules 

to Guide Remaining 

Efforts

Bureau Has Performed 

Many System Testing 

Activities, but Much 

Remains to be Done 

Table 2: Status of System Testing and Plans (Dress Rehearsal and 2010 Testing)

2010 system testing

System Dress Rehearsal system testing Testing status

Testing plan

completed

Testing schedule 

completed

Headquarters Processing—UC&M 

and RPS 

In progress In progress Partial Partial 

MAF/TIGER Completed In progress Partial Partial

DRIS Completed In progress PartialaPartiala

FDCA Partially completedbIn progress Partial Partial 

PBO N/AcIn progress No Partial 

DADS DADSd in progress DADS II in 

progress 

Partial Partial

Source: GAO analysis of Bureau data. 

a

Program officials stated that DRIS’s test plan and schedule were completed but will be modified to 

reflect changes resulting from the switch to paper-based operations.

b

System testing related to operations removed from the Dress Rehearsal was not completed. These

operations were later moved to PBO. 

c

The office to support PBO was created in August 2008. 

d

DADS is being used for Dress Rehearsal system testing, but the replacement system, DADS II, is 

being developed and tested for 2010 operations. 

Page 11 GAO-09-262 Census Testing
 

For both Headquarters Processing Systems (UC&M and RPS), system 

testing for the Dress Rehearsal has been partially completed, as shown in 

table 3. 

Dress Rehearsal System

Testing for Headquarters 

Processing Systems Is 

Partially Completed, but 

Additional Test Plans for 

2010 System Testing Have 

Not Yet Been Developed 

• For UC&M, Dress Rehearsal system testing is divided into three phases, as 

shown. These phases include a total of 19 products (used to control and 

track Dress Rehearsal enumeration activities). The completed phases (1 

and 2) included the development and testing of 14 products. Program 

officials had planned to complete testing of the remaining 5 products for 

UC&M by October 2008, but as of December 2008, the program had not yet 

completed this testing. 

• For RPS, Dress Rehearsal system testing is being done by component—

eight components perform functions for key activities such as data 

integration—where response data are integrated before processing. 

According to program officials, development and testing of four 

components are complete, and the remaining four components are 

planned to be completed by March 2009. 

Table 3: Status of System Testing and Plans for Headquarters Processing Systems (Dress Rehearsal and 2010 Testing)

Dress Rehearsal system testing 2010 system testing

System Dates

Testing 

status Dates Testing status

Testing plan

completed

Testing schedule 

completed

UC&M Partial Partiala

Phase 1 7/07–9/07 Completed 6/09–8/09 Not started

Phase 2 12/07–5/08 Completed 10/09–5/10 Not started

Phase 3 7/08– In progressb6/10–8/10 Not started

RPS Partial Partiala

Components 1–4 1/08–9/08 Completed 12/09–9/10 Not startedc

Components 5–8 10/08– In progressb 9/10–12/10 Not started 

Source: GAO analysis of Bureau data. 

a

High-level schedules have been defined; detailed schedules are not complete. 

b

Completion has been delayed. 

c

For 2010 operations, only two components will be included; two were combined and one was

omitted. 

In addition to ongoing Dress Rehearsal system testing, the program office

intends to perform system testing for 2010 census operations, but plans for 

this testing have not yet been developed. According to program officials, 

they have not developed testing plans and schedules for additional testing 

for the 2010 census because Bureau management has not yet finalized the 

requirements for 2010 operations. Finalizing these requirements may 

Page 12 GAO-09-262 Census Testing
 

involve both changes to existing requirements and new requirements. 

Program officials stated that they do not anticipate substantial changes in 

UC&M and RPS system requirements for 2010 census operations and plan

to have them finalized by May 2009. In commenting on a draft of this 

report, the Bureau provided an initial test plan and schedule, but did not 

provide the finalized baselined requirements for the Headquarters 

Processing Systems for 2010 operations. 

Until the baseline requirements are established, it is unclear whether the 

amount of additional testing necessary for 2010 census operations will be 

significant. According to industry best practices, defining requirements for a 

system is important because they provide a baseline for development and 

testing activities and are used to establish test plans, which define schedule

activities, roles and responsibilities, resources, and system testing priorities. 

The absence of finalized requirements increases the risk that there may not be

sufficient time and resources to adequately test the systems, which are critical

to ensuring that address files are accurately organized into enumeration 

universes and that duplicate responses are eliminated. 

System testing has been partially completed for MAF/TIGER products 

(that is, extracts from the MAF/TIGER system) required for the 2010 

census. For MAF/TIGER, testing activities are defined by products needed

for key activities, such as address canvassing. During Dress Rehearsal 

system testing, the program office completed testing for a subset of 

MAF/TIGER products for address canvassing, group quarters validation,

and other activities. 

MAF/TIGER Program Has 

Partially Completed Testing, 

but Test Plans and Schedules 

Are Incomplete and Ability to 

Track Progress Is Unclear 

Additional system testing is planned for the 2010 census. According to 

program officials, as of December 2008, the Bureau had defined 

requirements and completed testing for 6 of approximately 60 products 

needed for 2010 operations (these 6 products are related to address 

canvassing). The program office has also developed detailed test plans and 

schedules through April 2009, but these do not cover all of the remaining 

products needed to support the 2010 census. Table 4 is a summary of the 

status of MAF/TIGER 2010 testing and plans. 

Page 13 GAO-09-262 Census Testing
 

Table 4: Status of System Testinga and Plans for MAF/TIGER (2010 Testing)

Census activities Dates 

Testing 

status

Testing plan

completed

Testing 

schedule 

completed

Address canvassing 

(including testing of 

MAF/TIGER updates) 

5/08–5/09 In progress Partial Partial 

Group quarters validation 12/08–9/09 In progress Partial Partial

Enumeration universe 

(including group quarters 

enumeration, enumeration 

of transitory locations) 

12/08–10/10 In progress Partial Partial

Nonresponse follow-up 3/09–6/10 Not started Partial Partial

Field verification 10/08–10/10 In progress Partial Partial

Post-census products 3/10–11/10 Not started Partial Partial

Source: GAO analysis of Bureau data. 

a

System tests of a subset of MAF/TIGER products were also performed during the Dress Rehearsal.

According to program officials, the detailed test plans for the remaining

products will be developed after the requirements for each are finalized. As 

mentioned, establishing defined requirements is important because these 

provide a baseline for development and testing activities and define the basic 

functions of a product. The officials stated that they were estimating the

number of products needed, but would only know the exact number when

the requirements for the 2010 census operations are determined. The officials

added that, by January 2009, they plan to have a detailed schedule and a list of

the products needed through December 2009. 

Without knowing the total number of products, related requirements, and 

when the products are needed for operations, the Bureau risks both not 

sufficiently defining what each product needs to do and not being able to

effectively measure the progress of MAF/TIGER testing activities, which 

therefore increases the risk that there may not be sufficient time and 

resources to adequately test the system and that the system may not 

perform as intended. 

Page 14 GAO-09-262 Census Testing
 

System testing has been partially completed for DRIS components, 

including paper, workflow control and management, and telephony.14

Portions of the functionality in each DRIS component are being developed 

and tested across five increments for 2010 operations. As of November 

2008, the program had planned and completed testing of increments 1, 2, 

and 3. Testing of increment 4 is currently ongoing. Table 5 is a summary of 

the status of DRIS testing for 2010 operations. (In addition, system testing 

of a subset of DRIS functionality, including the integration of certain 

response data, took place during the Dress Rehearsal.) 

DRIS Testing Is Under Way; 

Plans Have Been Established 

but Will Be Revised 

Table 5: Status of System Testinga and Plans for DRIS (2010 Testing)

Phase Dates Testing status

Testing plan

completedb

Testing schedule 

completedb

Increment 1 12/07–3/08 Completed N/A N/A

Increment 2 3/08–7/08 Completed N/A N/A

Increment 3 6/08–11/08 Completed N/A N/A

Increment 4 10/08–5/09 In progress Partial Partial

Increment 5 3/09–7/09 Not started Partial Partial

Source: GAO analysis of Bureau data. 

a

System tests of a subset of DRIS functionality were also performed during the Dress Rehearsal. 

b

Program officials stated that DRIS’s test plan and schedule were completed, but will be modified to 

reflect changes resulting from the switch to paper-based operations.

The DRIS program has developed a detailed testing plan and schedule, 

including the remaining testing for increment 5. For example, detailed 

testing plans have been developed for all 558 functional requirements for 

DRIS. According to program officials, most of the 558 functional 

requirements will be fully tested during increments 4 and 5. As of 

November 2008, 22 of the 558 requirements had been tested.

Although plans and schedules were completed, the change from handheld

computers to paper processes for nonresponse follow-up has caused 

changes to DRIS processing requirements. For example, DRIS program 

officials stated that they now need to process an additional 40 million 

 14DRIS functionality includes the following: (1) the paper segment processes census paper

forms; (2) the workflow control and management segment provides the databases, 

workflow, and interfaces to capture response data, store these data, and transfer data 

between segments and external entities; and (3) the telephony segment provides 

infrastructure and application for performing coverage follow-up operations, telephone 

questionnaire assistance, and interactive voice response operations. 

Page 15 GAO-09-262 Census Testing
 

paper forms generated as a result of this switch. Although DRIS program 

officials stated that they are prepared to adjust their test schedule and plan 

to accommodate this change, they cannot do so until the requirements 

have been finalized for the switch to paper processes. (This responsibility 

is primarily that of the PBO program office.) Furthermore, based on the 

switch to paper, DRIS may not be able to conduct a test using these 

operational systems and live data. This increases the risk that the Bureau

could experience problems with these systems and the processing of 

paper forms during the 2010 census. The DRIS program office is 

addressing this risk by developing alternative strategies for testing and 

providing additional resources as contingencies for activities that may not

be fully tested before 2010 operations begin. 

FDCA testing has been partially completed, but much more work remains. 

System testing for FDCA took place during the Dress Rehearsal, but 

problems encountered during this testing led to the removal of key 

operations from the Dress Rehearsal15 and the April 2008 redesign, as 

described earlier. Going forward, FDCA development and testing for 2010 

operations are being organized based on key census activities. For 

example, FDCA testing for the address canvassing and group quarters 

validation operations was completed in December 2008. The FDCA 

contractor is currently developing and testing a backup system (known as

the continuity of operations system) for address canvassing and group 

quarters validation, and is currently testing another system (known as map 

printing) to provide printed maps for paper-based field operations, such as 

nonresponse follow-up. Table 6 summarizes the FDCA test status. 

FDCA System Testing Is 

Proceeding on an Aggressive 

Schedule, but Key Test Plans 

Have Not Been Developed 

Table 6: Status of System Testing and Plans for FDCA (2010 Testing) 

Census activity Dates Status

Testing 

plan

completed

Testing 

schedule 

completed

Address canvassing/group 

quarters validation 

5/08–12/08 Partially

completed 

N/A N/A

Map printing 5/08–6/09 In progress Partial Partial

Continuity of operations 9/08–5/09 In progress Partial Partial

Source: GAO analysis of Bureau data. 

 15These include update/leave, nonresponse follow-up, enumeration of transitory locations, 

group quarters enumeration, and field verification, as mentioned earlier. 

Page 16 GAO-09-262 Census Testing
 

Although system testing for address canvassing and group quarters 

validation was recently completed, program officials have not 

demonstrated that all current system requirements have been fully tested. 

As part of a contract revision required by the April 2008 redesign, the 

FDCA program established a revised baseline for system requirements on 

November 20, 2008. According to program officials, this revision included 

both modifications to previous requirements and removal of requirements 

that were part of activities transferred to PBO. As of December 2008,

program officials stated that detailed testing plans for many of the 

requirements exist, but need to be revised to address the newly baselined

requirements. 

Furthermore, as of December 2008, the FDCA program had not finalized 

detailed testing plans and schedules for the continuity of operations and 

map printing systems. According to program officials, they had not yet 

developed detailed testing plans and schedules for testing systems’ 

requirements because their focus has been on testing the system for 

address canvassing and group quarters validation. Officials added that they 

plan to begin testing the requirements for the continuity of operations 

system in January 2009, and for the map printing system in February 2009. 

However, without having established testing plans and schedules for these 

systems, it is unclear what amount of testing will be needed and whether 

sufficient time has been allocated in the schedule to fully test these 

systems before they are needed for operations. 

Testing has only recently started for PBO because it is still in the 

preliminary phase of program planning and initial system development as 

the Bureau shifts responsibility for certain operations from the FDCA 

program office to PBO. Because this office has only recently been created, 

it is currently hiring staff, developing a schedule for several iterations of 

development, establishing a means to store and trace requirements, 

developing testing plans, and establishing a configuration management 

process. 

Paper-Based Operations Are in 

Early Development, but 

Detailed 2010 Testing Plans and 

Schedules Have Not Yet Been 

Developed 

According to program officials, development will occur in five releases, 

numbered 0 through 4. The first release, Release 0, is planned to contain 

functionality for the nonresponse follow-up and group quarters 

enumeration activities. Table 7 provides the current status of PBO Release 

0 test activities. 

Page 17 GAO-09-262 Census Testing
 

Table 7: Status of System Testing and Plans for PBO Release 0 (2010 Testing)

Phase Dates Status

Testing 

plan

completed

Testing 

schedule 

completed

Complete initial system testing 

(Alpha testing)

10/08–3/09 Started No Partial

Additional testing (Beta testing) 3/09–4/09 Not started No Partial

Mock field testing activities for 

nonresponse follow-up and group 

quarters enumeration 

4/09–5/09 Not started No Partial

Source: GAO analysis of Bureau data. 

However, the Bureau still has not yet determined when detailed testing 

plans and schedules for PBO systems will be developed. Officials stated 

that a more detailed schedule for Release 0 and development schedules for 

the remaining releases are under development, and that they plan to have 

the majority of the schedules developed by the end of January 2009. In 

commenting on a draft of this report, the Bureau provided a partial 

schedule for PBO test activities. 

Furthermore, officials stated they had not yet fully defined which 

requirements PBO would be accountable for, and which of these

requirements will be addressed in each iteration of development. The officials 

did state that the requirements will be based on those requirements

transferred from FDCA as part of the reorganization. Bureau officials stated

they had not yet completed these activities because responsibility for the 

requirements was only formally transferred as of October 2008. The program 

office expects to have its first iteration of requirements traceable to test cases 

by March 2009. However, officials did not know what percentage of program 

requirements will be included in this first iteration. 

Although progress has been made in establishing the PBO program office, 

numerous critical system development activities need to be planned and 

executed in a limited amount of time. Because of the compressed schedule 

and the large amount of planning that remains, PBO risks not having its 

systems developed and tested in time for the 2010 Decennial Census. 

Testing is critical to ensure that the paper forms used to enumerate 

residents of households who do not mail back their questionnaires, group 

quarters, and transitional living quarters are processed accurately. 

Page 18 GAO-09-262 Census Testing
 

The DADS system (which had been used in the 2000 census) is currently 

being tested during the Dress Rehearsal, which is scheduled to be 

completed in March 2009. However, the Bureau intends to replace DADS 

with DADS II, which is currently being developed and tested for 2010

operations. DADS II is still in the early part of its life cycle, and the 

program office has only recently started system testing activities. The two 

main DADS II components, the Replacement Tabulation System (RTS) and 

Replacement Dissemination System (RDS), are being developed and tested

across a series of three iterations. As of December 2008, the program had 

begun iterations 1 and 2 for RTS, and iteration 1 for RDS. Table 8 

summarizes the RTS and RDS testing status. 

DADS II Is in Early Stages of 

Development, and Testing 

Plans and Schedules Are Being 

Developed 

Table 8: Status of System Testing and Plans for DADS II Components (2010 

Testing) 

Phase Dates Status

Testing plan

completed

Testing schedule 

completed

RTS Partial

a

Partiala

Iteration 1 7/08–4/09 In progress

Iteration 2 9/08–10/09 In progress

Iteration 3 3/09–4/10 Not started

Deployment 2/10–7/10 Not started

RDS PartialaPartiala

Iteration 1 7/08–3/09 In progress

Iteration 2 1/09–3/10 Not started

Iteration 3 5/09–8/10 Not started

Deployment 7/10–2/11 Not started

Source: GAO analysis of Bureau data. 

a

High-level plans have been defined; detailed plans are not complete. 

The DADS II program office has developed a high-level test plan for RTS 

and RDS system testing, but has not yet defined detailed testing plans and 

a schedule for testing system requirements. System requirements for the 

new system have been baselined, with 202 requirements for RTS and 318

requirements for RDS. According to program officials, the program office

is planning to develop detailed testing plans for the system requirements 

for both RTS and RDS. 

Page 19 GAO-09-262 Census Testing
 

Effective integration testing ensures that external interfaces work 

correctly and that the integrated systems meet specified requirements. 

This testing should be planned and scheduled in a disciplined fashion 

according to defined priorities. 

For the 2010 census, each program office is responsible for and has made 

progress in defining system interfaces and conducting integration testing, 

which includes testing of these interfaces. However, significant activities 

remain in order for comprehensive integration testing to be completed by

the date that the systems are needed for 2010 operations. For example, 

DRIS has conducted integration testing with some systems, such as FDCA, 

UC&M, and RPS, and is scheduled to complete integration testing by 

February 2010. The FDCA program office has also tested interfaces related 

to the address canvassing operation scheduled to begin in April 2009. 

However, for many other systems, such as PBO, interfaces have not been

fully defined, and other interfaces have been defined but have not been 

tested. Table 9 provides the status of integration testing among key 

systems. 

Bureau Has Conducted 

Limited Integration 

Testing, but Has Not 

Developed 2010 Test Plans 

and Schedules for 

Integration Testing 

Table 9: Integration Testing Status

System

Testing 

status Description

UC&M and 

RPS 

In progress Conducted testing during the Dress Rehearsal, but until 

requirements are baselined for the 2010 census, it is 

unclear whether changes have occurred that would require

retesting. According to program officials, all UC&M and RPS 

interfaces will be fully tested by an independent testing 

group within the Bureau. 

MAF/TIGER In progress Conducted testing with systems, such as FDCA, during the 

Dress Rehearsal and in preparation for the 2010 address 

canvassing operation. However, interfaces with other 

systems, such as RPS and UC&M, are still in development

and have not been retested following the Dress Rehearsal. 

DRIS In progress Conducted testing with systems, such as FDCA, for the 

group quarters validation operation. However, interfaces 

with other systems, such as PBO, are still in development

and have not been tested. According to program officials, 

interface testing may not be completed until February 2010 

due to the limited availability of other systems for testing. 

FDCA In progress Conducted testing with systems, such as DRIS and 

MAF/TIGER, for the address canvassing and group quarters 

validation operations. However, interface testing with 

systems for other operations, such as map printing, has not

been completed. 

Page 20 GAO-09-262 Census Testing
 

System

Testing 

status Description

PBO Not started Evaluating interfaces but has not yet fully defined or 

developed them. 

DADS II Not started Defining and developing its interfaces for the RDS and RTS

system. According to program documentation, interfaces for 

RTS are planned to be fully tested by July 2009, and 

interfaces for RDS are planned to be fully tested by July 

2010. 

Source: GAO analysis of Bureau data. 

In addition, the Bureau has not established a master list of interfaces 

between key systems, or plans and schedules for integration testing of 

these interfaces. A master list of system interfaces is an important tool for 

ensuring that all interfaces are tested appropriately and that the priorities 

for testing are set correctly. Although the Bureau had established a list of

interfaces in 2007, according to Bureau officials, it was not updated 

because of resource limitations at the time and other management 

priorities. As of October 2008, the Bureau had begun efforts to update this

list, but it has not provided a date when this list will be completed. 

Without a completed master list, the Bureau cannot develop

comprehensive plans and schedules for conducting systems integration 

testing that indicate how the testing of these interfaces will be prioritized. 

This is important because a prioritized master list of system interfaces, 

combined with comprehensive plans and schedules to test the interfaces, 

would allow for tracking the progress of this testing. With the limited 

amount of time remaining before systems are needed for 2010 operations, 

the lack of comprehensive plans and schedules increases the risk that the

Bureau may not be able to adequately test system interfaces, and that 

interfaced systems may not work together as intended. 

Bureau Has Conducted 

Limited End-to-End 

Testing as Part of the 

Dress Rehearsal, but Has 

Not Developed Testing 

Plans for Critical 

Operations 

The Dress Rehearsal was originally conceived to provide a comprehensive 

end-to-end test of key 2010 census operations; however, as mentioned 

earlier, because of the problems encountered with the handheld devices, 

among other things, testing was curtailed. As a result, although several 

critical operations underwent end-to-end testing in the Dress Rehearsal, 

others did not. According to the Associate Director for the 2010 census, 

the Bureau tested approximately 23 of 44 key operations during the Dress 

Rehearsal. Examples of key operations that underwent end-to-end testing 

during the Dress Rehearsal are address canvassing and group quarters 

validation. An example of a key operation that was not tested is the largest 

field operation—nonresponse follow-up. 

Page 21 GAO-09-262 Census Testing
 

Although the Bureau recently conducted additional testing of the handhelds,

this test was not a robust end-to-end test. In December 2008, after additional 

development and improvements to the handheld computers, the Bureau

conducted a limited field test for address canvassing, intended to assess

software functionality in an operational environment. We observed this test 

and determined that users were generally satisfied with the performance of

the handhelds. According to Bureau officials, the performance of the 

handheld computers has substantially improved from previous tests.

However, the test was not designed to test all the functionality of the 

handhelds in a robust end-to-end test—rather, it included only a limited

subset of functionality to be used during the 2009 address canvassing 

operations. Further, the field test did not validate that the FDCA system fully 

met specified requirements for the address canvassing operation. Bureau

officials stated that additional testing of the FDCA system, such as

performance testing, mitigated the limitations of this field test. 

Nonetheless, the lack of robustness of the field test poses several risks for 

2010 operations. Specifically, without testing all the FDCA system’s 

requirements in a robust operational environment, it is unclear whether 

the system can perform as intended when the address canvassing 

operation begins in April 2009. 

Furthermore, as of December 2008, the Bureau has neither established 

testing plans nor schedules to perform end-to-end testing of the key 

operations that were removed from the Dress Rehearsal, nor has it 

determined when these plans will be completed. As previously mentioned, 

these operations include 

• update/leave, 

• nonresponse follow-up, 

• enumeration of transitory locations,

• group quarters enumeration, and 

• field verification. 

Although the Bureau has established a high-level strategy for testing these 

operations, which provides details about the operations to be tested, 

Bureau officials stated that they have not developed testing plans and 

schedules because they are giving priority to tests for operations that are 

needed in early 2009. In addition, key systems needed to test these 

operations are not ready to be tested because they are either still in 

development or have not completed system testing. Until system and 

Page 22 GAO-09-262 Census Testing
 

integration testing activities are complete, the Bureau cannot effectively 

plan and schedule end-to-end testing activities. Without sufficient end-toend testing, operational problems can go undiscovered, and the 

opportunity to improve these operations will be lost. 

The decreasing time available for completing end-to-end testing increases

the risk that testing of key operations will not take place before the 

required deadline. Bureau officials have acknowledged this risk in 

briefings to the Office of Management and Budget. The Bureau is in the 

process of identifying risks associated with incomplete testing and 

developing mitigation plans, which it had planned to have completed by 

November 2008. However, as of January 2009, the Bureau had not 

completed these mitigation plans. According to the Bureau, the plans are 

still being reviewed by senior management. Without plans to mitigate the 

risks associated with limited end-to-end testing, the Bureau may not be

able to respond effectively if systems do not perform as intended. 

As stated in our testing guide and IEEE standards, oversight of testing 

activities includes both planning and ongoing monitoring of testing 

activities. Ongoing monitoring entails collecting and assessing status and 

progress reports to determine, for example, whether specific test activities 

are on schedule. Using this information, management can effectively 

determine whether corrective action is needed and, if so, what action 

should be taken. In addition, comprehensive guidance should describe 

each level of testing (for example, system, integration, or end-to-end), 

criteria for each level, and the type of test products expected. The 

guidance should also address test preparation and oversight activities. 

Although the 2010 Decennial Census is managed by the Decennial 

Management Division, the oversight and management of key census IT 

systems is performed on a decentralized basis. DRIS, FDCA, and DADS II 

each have a separate program office within the Decennial Automation 

Contracts Management Office; Headquarters Processing and PBO are 

managed within the Decennial System and Processing Office; and 

MAF/TIGER is managed within the Geography Division. Each program has 

its own program management reviews and develops plans and tracks 

metrics related to testing. These offices and divisions collectively report to

the Associate Director for the 2010 Census. According to the Bureau, the 

associate director chairs biweekly meetings where the officials 

responsible for these systems meet to review the status of key systems 

development and testing efforts. 

Bureau Lacks Sufficient 

Executive-Level Oversight 

and Guidance for Testing 

Page 23 GAO-09-262 Census Testing
 

In addition, in response to prior recommendations, the Bureau took initial

steps to enhance its programwide oversight; however, these steps have not 

been sufficient to establish adequate executive-level oversight. In June 

2008, the Bureau established an inventory of all testing activities specific 

to all key decennial operations. This inventory showed that, as of May 

2008, 18 percent of about 1049 system testing activities had not been 

planned. (See fig. 4.) In addition, approximately 67 percent of about 836 

operational testing activities had not been planned. 

Figure 4: Inventory of Testing Activities as of May 2008

Although officials from the Decennial System and Processing Office 

described the inventory effort as a means of improving executive-level 

oversight, the inventory has not been updated since May 2008, and officials 

have no plans for further updates. Instead, officials stated that they plan to 

track testing progress as part of the Bureau’s detailed master schedule of 

census activities. However, this schedule does not provide comprehensive 

status information on testing. 

In another effort to improve executive-level oversight, the Decennial 

Management Division began producing (as of July 2008) a weekly 

Source: GAO analysis of Bureau data. 

System testing Operational testing

1%

73%

18%

7% 6%

25%

67%

3%

Testing complete

Testing planned or partially complete

No testing planned

Testing status unknown

Page 24 GAO-09-262 Census Testing
 

executive alert report and has established (as of October 2008) monthly 

dashboard and reporting indicators. However, these products do not 

provide comprehensive status information on the testing progress of key 

systems and interfaces. For example, the executive alert report does not 

include the progress of testing activities, and although the dashboard 

provides a high-level, qualitative assessment of testing for key operations

and selected systems, it does not provide information on the testing 

progress of all key systems and interfaces. 

Further, the assessment of testing progress has not been based on 

quantitative and specific metrics. For example, the status of testing key 

operations removed from the Dress Rehearsal was marked as acceptable,

or “green,” although the Bureau does not yet have plans for testing these 

activities. Bureau officials stated that they marked these activities as 

acceptable because, based on past experience, they felt comfortable that a 

plan would be developed in time to adequately test these operations. The 

lack of quantitative and specific metrics to track progress limits the 

Bureau’s ability to accurately assess the status and progress of testing 

activities. In commenting on a draft of this report, the Bureau provided 

selected examples in which they had begun to use more detailed metrics 

to track the progress of end-to-end testing activities. 

Finally, although the Bureau announced in August 2008 that it was 

planning to hire a senior manager who would have primary responsibility 

for monitoring testing across all decennial systems and programs, the 

position had not been filled as of January 2009. Instead, agency officials 

stated that the role is being filled by another manager from the Decennial 

Statistical Studies Division, who has numerous other responsibilities. 

The Bureau also has weaknesses in its testing guidance; it has not 

established comprehensive guidance for system testing. According to the 

Associate Director for the 2010 Census, the Bureau did establish a policy 

strongly encouraging offices responsible for decennial systems to use best 

practices in software development and testing, as specified in level 2 of 

Carnegie Mellon’s Capability Maturity Model®

 Integration.16 However, 

beyond this general guidance, there is no additional guidance on key 

testing activities such as criteria for each level or the type of test products 

 16Capability Maturity Model®

 Integration is intended to provide guidance for improving an

organization’s processes, and gives the ability to manage the development, acquisition, and 

maintenance of products and services. The model uses capability levels to assess process 

maturity. 

Page 25 GAO-09-262 Census Testing
 

expected. Standardized policies and procedures help to ensure 

comprehensive processes across an organization and allow for effective 

executive-level oversight. The lack of guidance has led to an ad hoc—and, 

at times—less than desirable approach to testing.

While the Bureau’s program offices have made progress in testing key 

decennial systems, much work remains to ensure that systems operate as 

intended for conducting an accurate and timely 2010 census. Several 

program offices have yet to prepare and execute system test plans and 

schedules and ensure that system requirements are fully tested. In 

addition, the Bureau has not developed a master list of interfaces, which is

necessary to prioritize testing and to develop comprehensive integration

test plans and schedules. Additionally, end-to-end testing plans for key 

operations have not been finalized or executed based on established 

priorities to help ensure that systems will support census operations. 

Weaknesses in the Bureau’s IT testing can be attributed, in part, to a lack

of sufficient executive-level oversight and guidance. More detailed metrics 

and status reports would help the Bureau to better monitor testing 

progress and identify and address problems. Giving accountability for 

testing to a senior-level official would also provide the focus and attention 

needed to complete critical testing. Also, completing risk mitigation plans 

will help ensure that actions are in place to address potential problems 

with systems. Given the rapidly approaching deadlines of the 2010 census,

completing these important tests and establishing stronger executive-level

oversight and guidance are critical to ensuring that systems perform as 

intended when they are needed. 

To ensure that testing activities for key systems for the 2010 census are 

completed, we are making 10 recommendations. We recommend that the

Secretary of Commerce require the Director of the Census Bureau to 

expeditiously implement the following recommendations: 

Conclusions 

Recommendations for 

Executive Action 

• For the Headquarters UC&M and RPS, finalize requirements for 2010

census operations and complete testing plans and schedules for 2010 

operations that trace to baselined system requirements. 

• For MAF/TIGER, establish the number of products required, define related 

requirements, and establish a testing plan and schedule for 2010 

operations. 

Page 26 GAO-09-262 Census Testing
 

• For FDCA, establish testing plans for the continuity of operations and map 

printing systems that trace to baselined system requirements. 

• For PBO, develop baseline requirements and complete testing plans and

schedules for 2010 operations. 

• Establish a master list of system interfaces; prioritize the list, based on 

system criticality and need date; define all interfaces; and develop 

integration testing plans and schedules for tracking the progress of testing 

these interfaces. 

• Establish a date for completing testing plans for the operations removed 

from the Dress Rehearsal operations and prioritize testing activities for 

these operations. 

• Finalize risk mitigation plans detailing actions to address system problems 

that are identified during testing. 

• Establish specific testing metrics and detailed status reports to monitor 

testing progress and better determine whether corrective action is needed

for all key testing activities. 

• Designate a senior manager with primary responsibility for monitoring 

testing and overseeing testing across the Bureau. 

• In addition, after the 2010 census, we recommend that the Bureau 

establish comprehensive systems and integration testing guidance to guide 

future testing of systems. 

The Associate Under Secretary for Management of the Department of 

Commerce provided written comments on a draft of this report. The 

department’s letter and general comments are reprinted in appendix II. 

In the comments, the department and Bureau stated they had no 

significant disagreements with our recommendations. However, the 

department and Bureau added that since the FDCA replan last year, their

testing strategy has been to focus on those things they have not done 

before, and to demonstrate to their own satisfaction that new software and 

systems will work in production. The department added that it has 

successfully conducted Census operations before, and was focusing “on 

testing the new things for 2010—not things that have worked before.” 

While we acknowledge that the Bureau has conducted key census 

operations before, the systems and infrastructure in place to conduct 

these operations have changed substantially since the 2000 census. For 

example, while the Bureau has conducted paper-based nonresponse 

Agency Comments 

and Our Evaluation 

Page 27 GAO-09-262 Census Testing
 

followup during previous censuses, it will be using newly developed 

systems which have not yet been fully tested in a census-like environment 

to integrate responses and manage the nonresponse followup work load. 

In addition, new procedures, such as one to remove questionnaires that 

were mailed in late from the nonresponse followup operation, have not 

been tested with these systems. Any significant change to an existing IT 

system introduces the risk that the system may not work as intended; 

therefore, testing all systems after changes have been made to ensure the

systems work as intended is critical to the success of the 2010 census. 

In addition, the department and Bureau provided technical comments, 

such as noting draft plans that had been developed after the conclusion of 

our work, that we have incorporated where appropriate. 

We are sending copies of this report to the Secretary of Commerce, the 

Director of the U.S. Census Bureau, and other appropriate congressional 

committees. The report also is available at no charge on the GAO Web site 

at http://www.gao.gov. If you have any questions about this report, please 

contact David Powner at (202) 512-9286 or pownerd@gao.gov. GAO staff 

who made contributions to this report are listed in appendix III. Contact 

points for our Offices of Congressional Relations and Public Affairs may 

David A. Powner 

be found on the last page of this report. 

s

Director, Information Technology 

Management Issue

Page 28 GAO-09-262 Census Testing
 

List of Congressional Requesters 

The Honorable Tom Carper 

Chairman 

Subcommittee on Federal Financial Management, 

 Government Information, Federal Services, and 

 International Security 

Committee on Homeland Security and 

 Governmental Affairs 

United States Senate 

The Honorable Edolphus Towns 

Chairman 

The Honorable Darrell Issa 

Ranking Member 

Committee on Oversight and Government Reform 

House of Representatives 

The Honorable Wm. Lacy Clay 

Chairman 

The Honorable Patrick T. McHenry 

Ranking Member 

Subcommittee on Information Policy, 

 Census, and National Archives 

Committee on Oversight and Government Reform 

House of Representatives 

The Honorable Michael R. Turner 

House of Representatives 

Page 29 GAO-09-262 Census Testing
Appendix I: Scope and Methodology Appendix I: Scope and Methodology 

To determine the status of and plans for testing key decennial systems, we

analyzed documentation related to system, integration, and end-to-end 

testing.1

 For system testing, we analyzed documentation related to each 

key decennial system, including system test plans, schedules,

requirements, results, and other test-related documents. We then 

compared the Bureau’s practices with those identified in our testing guide 

and Institute of Electrical and Electronics Engineers (IEEE) standards2

 to 

determine the extent to which the Bureau had incorporated best practices 

in testing. We also interviewed program officials and contractors of key 

decennial systems to obtain information on the current status of and plans 

for testing activities. 

For integration testing, we analyzed interface control documents, interface 

testing plans, and schedules. We also analyzed documentation of the 

Census Bureau’s (Bureau) oversight of integration testing activities, 

including efforts to issue integration testing guidance and monitor the 

progress of integration testing activities. We interviewed program officials 

at each key decennial system program office to obtain information on the 

current status of and plans for integration testing and interviewed program 

officials at the Decennial Systems Processing Office to obtain information 

on the executive-level oversight of integration testing activities. We 

compared the Bureau’s practices with those identified in our testing guide 

and IEEE guidance. 

For end-to-end testing, we analyzed documentation related to the testing 

of key census operations during the Bureau’s Dress Rehearsal, additional 

testing conducted for the address canvassing operation, and efforts to 

establish testing plans and schedules for operations removed from the 

Dress Rehearsal. We also observed the Bureau’s operational field test, held

in December 2008 in Fayetteville, North Carolina. We interviewed program 

officials at the Decennial Systems Processing Office to obtain information 

on the current status and plans for end-to-end testing activities. We 

compared the Bureau’s practices with those identified in our testing guide 

and IEEE guidance. 

 1

System testing verifies that a system meets specified requirements. Integration testing 

verifies that systems, when combined, work as intended. End-to-end testing verifies that a 

set of systems work as intended in an operational environment. 

2

GAO/AIMD-10.1.21 and IEEE Std 12207-2008. 

Page 30 GAO-09-262 Census Testing
Appendix I: Scope and Methodology 

We also analyzed documentation of the Bureau’s overall oversight of 

testing, including its executive alert reports and monthly dashboard 

reports. In addition, we assessed system testing guidance and interviewed 

the Associate Director for the Decennial Census to obtain information on 

the overall oversight of testing activities. 

We conducted this performance audit from June 2008 to February 2009 in 

the Washington, D.C., and Fayetteville, North Carolina, areas, in 

accordance with generally accepted government auditing standards. Those 

standards require that we plan and perform the audit to obtain sufficient, 

appropriate evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objective. We believe that the evidence 

obtained provides a reasonable basis for our findings and conclusions 

based on our audit objective. 

Page 31 GAO-09-262 Census Testing
A

of C

ppendix II: Comments from the Department 

ommerce 

Page 32 GAO-09-262

Appendix II: Comments from the Department 

of Commerce 

Census Testing
Appendix II: Comments from the Department 

of Commerce 

Page 33 GAO-09-262 Census Testing
Appendix III: GAO

A

 Contact and Staff 

cknowledgments 

Page 34 GAO-09-262

Appendix III: GAO Contact and Staff 

Acknowledgments 

David A. Powner, (202) 512-9286 or pownerd@gao.gov

In addition to the contact name above, individuals making contributions to 

this report included Cynthia Scott (Assistant Director), Sher`rie Bacon, 

Barbara Collier, Neil Doherty, Vijay D’Souza, Nancy Glover, Lee 

McCracken, Jonathan Ticehurst, Melissa Schermerhorn, and Karl Seifert.

Census Testing

GAO Contact 

Staff 

Acknowledgments 

(310881) 
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs