Report to the Honorable Jane Harman, 

House of Representatives 

United States General Accounting Office 

GAO 

November 2003 LAW ENFORCEMENT

Better Performance 

Measures Needed to 

Assess Results of 

Justice’s Office of 

Science and 

Technology 

GAO-04-198 
OST's budgetary resources grew significantly in recent years, along with the 

range of its program responsibilities. From fiscal year 1995 through fiscal 

year 2003, OST received over $1 billion through Department of Justice 

appropriations and the reimbursement of funds from other federal agencies 

in exchange for OST’s agreement to administer these agencies' projects. Of 

the over $1 billion that OST received, approximately $749 million, or 72 

percent, was either directed to specific recipients or projects by public law, 

subject to guidance in congressional committee reports, or directed though 

reimbursable agreements. At the same time that spending expanded, OST’s 

program responsibilities have changed—from primarily law enforcement 

and corrections to broader public safety technology. 

OST delivers three groups of products through various methods. The three 

groups include (1) information dissemination and technical assistance; (2) 

the application, evaluation, and demonstration of existing and new 

technologies for field users; and (3) technology research and development. 

According to OST, as of April 2003, it has delivered 945 products since its 

inception. Furthermore, OST identified an additional 500 products 

associated with ongoing awards. OST makes its products available through a 

variety of methods, such as posting information on its Web site and 

providing research prototypes to field users for testing and evaluation. 

OST has been unable to fully assess its performance in achieving its goals as 

required by applicable criteria because it does not use outcome measures to 

assess the extent to which it achieves the intended results of its programs. 

OST’s current measures primarily track outputs, the goods and services 

produced, or in some cases OST uses intermediate measures, which is a step 

toward developing outcome measures. The Government Performance and 

Results Act of 1993 provides that federal agencies measure or assess the 

results of each program activity. While developing outcome measures for the 

types of activities undertaken by OST is difficult, we have previously 

reported on various strategies that can be used to develop outcome 

measures, or, at least intermediate measures, for similar types of activities. 

OST’s annual budgetary resources in constant 2002 dollars, fiscal years 1995–2003 

Dollars in millions

1995 1996 1997 1998 1999 2000 2001 2002 2003

Fiscal year

Source: OST.

0

50

100

150

200

250

300

The mission of the Office of 

Science & Technology (OST), 

within the Department of Justice’s 

National Institute of Justice (NIJ), 

is to improve the safety and 

effectiveness of technology used by 

federal, state, and local law 

enforcement and other public 

safety agencies. Through NIJ, OST 

funds programs in forensic 

sciences, crime prevention, and 

standards and testing. To support 

these programs, Congress 

increased funding for OST from 

$13.2 million in 1995 to $204.2 

million in 2003 (in constant 2002 

dollars). GAO reviewed (1) the 

growth in OST’s budgetary 

resources and the changes in OST’s 

program responsibilities, (2) the 

types of products OST delivers and 

the methods used for delivering 

them; and (3) how well OST’s 

efforts to measure the success of 

its programs in achieving intended 

results meet applicable 

requirements. 

GAO recommends that the Director 

of NIJ reassess the measures used 

to evaluate OST’s progress toward 

achieving its goals and to better 

focus on outcome measures to 

assess results where possible. In 

those cases where measuring 

outcomes is, after careful 

consideration, deemed infeasible, 

we recommend developing 

appropriate intermediate measures 

that will help to discern program 

effectiveness. 

www.gao.gov/cgi-bin/getrpt?GAO-04-198.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Laurie Ekstrand 

at (202) 512-8777 or Ekstrandl@gao.gov. 

Highlights of GAO-04-198, a report to the 

Honorable Jane Harman, House of 

Representatives 

November 2003 

LAW ENFORCEMENT

Better Performance Measures Needed to 

Assess Results of Justice's Office of 

Science and Technology 
Page i GAO-04-198 Justice's Office of Science and Technology 

Letter 1

Results in Brief 2

Background 3

OST’s Budgetary Resources Have Grown and Program 

Responsibilities Have Changed 6

OST Delivers Three Groups of Products Through Various Methods 13

OST’s Performance Measurement Efforts Do Not Fully Meet 

Requirements 17

Conclusions 24

Recommendation 25

Agency Comments and Our Evaluation 25

Appendix I Scope and Methodology 28 

Appendix II Bugetary Resources for OST’s Programs in Current 

Year Dollars 32 

Appendix III OST’s 10 Categories of Products 

33 

Appendix IV OST’s Portfolio Areas 35 

Appendix V OST’s Operations 37 

Appendix VI OST’s Goals in its Fiscal Year 2004 Performance Plan 

and GAO’s Assessment 47 

Appendix VII Comments from the Department of Justice 51 

Contents 
Page ii GAO-04-198 Justice's Office of Science and Technology 

Appendix VIII GAO Contacts and Staff Acknowledgments 54 

GAO Contacts 54

Staff Acknowledgments 54

Tables 

Table 1: Flow of Budgetary Resources to OST’s Programs 7

Table 2: Budgetary Resources in Constant 2002 Dollars for OST’s 

Programs by NIJ Allocation, Fiscal Years 1995-2003 10

Table 3: Budgetary Resources in Constant 2002 Dollars for OST’s 

Investigative and Forensic Sciences by NIJ Allocation, 

Fiscal Years 1995-2003 13

Table 4: GAO’s Assessment of the 42 Measures OST Developed for 

11 of Its Initiatives 19

Table 5: OST’s Outside Studies of Its Initiatives 23

Table 6: Budgetary Resources in Current Dollars for OST’s 

Programs by NIJ Allocation, Fiscal Years 1995-2003 32

Table 7: GAO’s Groupings of OST’s Categories of Products and 

Examples of Each Category 33

Table 8: Total Funds Awarded for the Operations, Maintenance, 

and Technical Support of OST’s 10 Technology Centers, 

Fiscal Years 1995-2003 40

Table 9: OST’s Technology Centers, Their Affiliated Partners, and 

the Amounts Awarded to Support the Centers 42

Table 10: OST’s Performance Goals, Initiatives, and Measures for 

Fiscal Year 2004, and GAO’s Assessment 47

Figures 

Figure 1: OST’s Budgetary Resources in Constant 2002 Dollars, 

Fiscal Years 1995–2003 8

Figure 2: GAO’s Grouping of OST’s 945 Delivered Products, as of 

April 2003 14

Figure 3: OST’s Organizational Structure 38

Figure 4: OST’s 10 Technology Centers and the Regions They Serve 39

Figure 5: Stakeholders and Customers that Contribute to the 

Setting of OST’s priorities 44
Page iii GAO-04-198 Justice's Office of Science and Technology 

Abbreviations 

AAG Assistant Attorney General 

CITA Crime Identification Technology Act 

CLIP Crime Lab Improvement Program 

CODIS Combined DNA Index System 

COPS Community-Oriented Policing Services 

DNA deoxyribonucleic acid 

DOD Department of Defense 

FBI Federal Bureau of Investigation 

GAO General Accounting Office 

GPRA Government Performance and Results Act 

LLEBG Local Law Enforcement Block Grant 

NFSIA Paul Coverdell National Forensic Sciences Improvement 

Act 

NIJ National Institute of Justice 

NLECTC National Law Enforcement and Corrections Technology 

Centers 

OJP Office of Justice Programs 

OMB Office of Management and Budget 

OST Office of Science and Technology 

R&D research and development 

SSLEA State and Local Law Enforcement Assistance 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
Page 1 GAO-04-198 Justice's Office of Science and Technology 

November 14, 2003 

The Honorable Jane Harman 

House of Representatives 

Dear Ms. Harman: 

To enhance public safety and bring criminals to justice, it is important for 

law enforcement officials to benefit from the latest advances in science 

and technology. The mission of the Office of Science and Technology 

(OST), within the Department of Justice’s National Institute of Justice 

(NIJ), is to improve the safety and effectiveness of technology used by 

federal, state, and local law enforcement, corrections, and other public 

safety agencies. OST awards funds to research and develop more effective 

technology and improve access to technology in a wide range of areas. For 

example, OST funds programs in the areas of crime prevention 

technologies, investigative and forensic sciences, and electronic crime. 

Examples of products resulting from OST’s programs include a guide on 

school safety, an evaluation of police protective gear, a prototype for 

ground-penetrating radar, and a report on gunshot residue detection and 

interpretation. To support OST’s programs, Congress has significantly 

increased its funding, from $13.2 million in fiscal year 1995 to $204.2 

million in fiscal year 2003 (in constant 2002 dollars). 

In response to your interest about whether OST’s programs are achieving 

their intended results, we reviewed certain aspects of OST’s operations. 

Specifically, this report assesses (1) the growth in OST’s budgetary 

resources, from fiscal year 1995 to fiscal year 2003, and changes in OST’s 

program responsibilities; (2) what types of products OST delivers and the 

methods used to deliver these products to public safety agencies; and 

(3) how well OST’s efforts to measure the success of its programs in 

achieving intended results meet applicable requirements. 

To address our objectives, we collected and analyzed relevant data and 

reports and interviewed OST officials and NIJ officials, including NIJ 

executive staff and the Assistant NIJ Director for OST, division chiefs, and 

managers. We also collected data and interviewed officials at OST 

technology centers in Rockville, Maryland; and El Segundo and San Diego, 

California. Appendix I contains detailed information on the scope and 

methodology we used for this assessment. We conducted this engagement 

in accordance with generally accepted government auditing standards. 

United States General Accounting Office

Washington, DC 20548 
Page 2 GAO-04-198 Justice's Office of Science and Technology 

OST has grown in terms of both budgetary resources and the range of 

programs it operates.1

 From fiscal year 1995 through fiscal year 2003, OST 

received over $1 billion through several Department of Justice (Justice) 

appropriations accounts as well as the reimbursement of funds from other 

federal agencies in exchange for OST’s agreement to administer these 

agencies’ projects. Of the over $1 billion that OST has received, 

approximately $749.7 million, or about 72 percent, was either directed for 

specific recipients or projects by public law, subject to guidance in 

congressional committee reports designating specific recipients or 

projects, or directed from reimbursable agreements with other federal 

agencies for OST to manage their projects. At the same time that spending 

has expanded, OST’s program responsibilities have changed—from 

primarily law enforcement and corrections technologies to broader public 

safety technologies, including safe school initiatives. 

OST delivers three groups of products through various methods. The three 

groups include (1) information dissemination and technical assistance; 

(2) the application, evaluation, and demonstration of existing and new 

technologies for field users; and (3) technology research and development 

(R&D). According to OST, as of April 2003, it had delivered 945 products 

since its inception. Furthermore, OST identified an additional 500 products 

associated with ongoing awards. Depending on its research agenda, OST 

makes its products available through a variety of methods, such as posting 

information on its Web site and providing research prototypes to field 

users for testing and evaluation. While OST does not directly 

commercialize the results of its technology R&D, it does help link 

prototypes with potential developers. 

OST has been unable to fully assess its performance in achieving its goals 

because it does not measure the extent to which it achieves the intended 

outcomes of its programs. OST’s current measures primarily track outputs 

(goods and services produced). In some cases OST uses intermediate 

measures—a step closer to developing outcome measures—but has not 

taken this step toward better measurement in many cases where it may be 

 1

We are using “programs” to indicate the broad categories of OST’s individual projects. NIJ 

and OST have referred to these categories as both portfolio areas and programs. Our use of 

the term “programs” encompasses “portfolio areas” (see app. IV for OST’s portfolio areas) 

and the safe school technology, counterterrorism technology, and correction technology 

programs. NIJ and OST delineations between the various programs and various portfolio 

areas are flexible. For example, some of the projects to develop metal detectors and 

personnel locator devices would apply to both school safety technologies and corrections 

technologies programs and therefore could be placed in different portfolio areas. 

Results in Brief 
Page 3 GAO-04-198 Justice's Office of Science and Technology 

possible to do so. The Government Performance and Results Act of 1993 

(GPRA) provides, among other things, that federal agencies establish 

performance measures, including, the assessment of relevant outputs and 

outcomes of each program activity. Office of Management and Budget 

(OMB) guidance suggests that, to the extent possible, federal agencies 

measure or assess the extent to which they are achieving the intended 

outcomes of their programs. As part of Justice’s efforts to comply with 

GPRA, OST established goals and developed output, and some 

intermediate, measures to track its progress. While developing outcome 

measures for the types of activities undertaken by OST is difficult, we have 

previously reported on various strategies that can be used to develop 

outcome measures or at least intermediate measures for activities that are 

similar to those in OST’s portfolio of programs. 

So that OST does all that is possible to assess whether its programs are 

achieving their intended results, we are recommending that the Attorney 

General instruct the Director of NIJ to reassess OST’s performance 

measures to better focus on outcome measures. In commenting on a draft 

of this report, the Assistant Attorney General (AAG) for Justice’s Office of 

Justice Programs (OJP) agreed with our recommendation. The AAG made 

additional comments concerning the challenge of developing outcome 

measures for R&D activities, OST’s overall performance record, and the 

amount of OST’s funds that are directed for specific recipients and 

projects. We respond to these comments in the Agency Comments and 

Evaluation section of the report. OJP also provided technical comments, 

which have been incorporated in this report where appropriate. 

The Office of Science and Technology (OST) was created in fiscal year 

1995 following a long history of science and technology efforts within the 

National Institute of Justice (NIJ).2

 NIJ is a component of the Office of 

Justice Programs (OJP), a Justice agency that, among other things, 

provides assistance to state, tribal, and local governments. In establishing 

OST’s objectives and allocating funds for OST’s programs, the NIJ Director 

considers the priorities of many stakeholders, including the President, 

Congress, Justice, and state and local law enforcement and public safety 

agencies. 

 2

NIJ was established in statute by the Justice System Improvement Act of 1979 (P.L. 96-157, 

93 Stat. 1167 (1979)), which, among other things, amended the Omnibus Crime Control and 

Safe Streets Act of 1968 (P.L. 90-351, 82 Stat. 197 (1968)). 

Background 
Page 4 GAO-04-198 Justice's Office of Science and Technology 

In November 2002, Congress established OST and its mission and duties in 

statute as part of the Homeland Security Act of 2002 (the Act).3

 The Act 

specified OST’s mission “to serve as the national focal point for work on 

law enforcement technology; and to carry out programs that, through the 

provision of equipment, training, and technical assistance, improve the 

safety and effectiveness of law enforcement technology and improve 

access to such technology by federal, state, and local law enforcement 

agencies.” The Act defined the term “law enforcement technology” to 

include “investigative and forensic technologies, corrections technologies, 

and technologies that support the judicial process.”4

 The Act also specified 

OST’s duties to include the following, among others: 

• establishing and maintaining advisory groups to assess federal, state, and 

local technology needs; 

• establishing and maintaining performance standards, and testing, 

evaluating, certifying, validating, and marketing products that conform to 

those standards; 

• carrying out research, development, testing, evaluation, and cost-benefit 

analysis of certain technologies; and 

• developing and disseminating technical assistance and training materials. 

OST’s operations have multiple levels of internal organization and multiple 

kinds of external partners. (For a more detailed description of OST’s 

operations, see app. V.) OST’s multiple levels of organization include a 

Washington, D.C., office and a network of 10 technology centers that 

provide technical assistance to OST’s customers around the country.5

 To 

fulfill its mission, OST also collaborates with entities such as the 

 3

P.L. 107-296, 116 Stat. 2135, 2159 (2002). These mission and duties are not unlike what OST 

had been carrying out previously. The Act codified the mission and duties in statute. 

4

According to NIJ, forensic science is the application of established scientific techniques to 

the identification, collection, and examination of evidence from crime scenes; the 

interpretation of laboratory findings; and the presentation of reported findings in judicial 

proceedings. 

5

These 10 technology centers are OST’s National Law Enforcement and Corrections 

Technology Center (NLECTC) system. 

OST Established in Statute 

by the Homeland Security 

Act of 2002 

OST’s Operations 
Page 5 GAO-04-198 Justice's Office of Science and Technology 

Departments of Defense and Energy and public and private laboratories to 

take advantage of established technical expertise and resources. 

NIJ has three main types of awards for funding OST’s programs: grants, 

interagency agreements, and cooperative agreements.6

• Grants are generally awarded annually by NIJ to state and local agencies 

or private organizations for a specific product and amount. 

• Interagency agreements are used by NIJ for creating partnerships with 

federal agencies. 

• Cooperative agreements are a type of NIJ grant to nonfederal entities that 

prescribes a higher level of monitoring and federal involvement. 

NIJ also uses memorandums of understanding (MOU) to coordinate 

programs and projects between agencies. The MOUs specify the roles, 

responsibilities, and funding amounts to be provided by participating 

agencies. Through NIJ, OST can provide supplemental funding to 

interagency and cooperative agreements that may be used to contract for 

special projects. 

OST awards are administered by managers at its Washington, D.C., office 

who have final oversight and management responsibility. These managers 

may delegate some responsibility to another federal R&D agency receiving 

the award. In March 2003, 21 managers were responsible for overseeing 

336 active awards totaling $636 million. 

Guidance has been established for measuring the performance of 

government operations. To assist Justice to follow the Government 

Performance and Results Act of 1993 (GPRA),7

 OST establishes goals and 

develops performance measures to track its progress. In addition, in May 

2002, the White House Office of Management and Budget (OMB) and 

Office of Science and Technology Policy issued a memorandum setting 

forth R&D investment criteria that departments and agencies should 

implement. The investment criteria require an explanation of why the 

investment is important, how funds will be allocated to ensure quality, and 

 6

We did not include contracts because NIJ uses them for the purchase of goods and 

services rather than for awarding funds for carrying out OST programs and projects. 

7

P.L.103-62, 107 Stat. 285 (1993). 
Page 6 GAO-04-198 Justice's Office of Science and Technology 

how well the investment is performing. According to the memorandum, 

program managers must define appropriate outcome measures, and 

milestones that can be used to track progress toward goals and assess 

whether funding should be enhanced or redirected. The memorandum 

encourages federal R&D agencies to make the processes they use to 

satisfy GPRA consistent with these criteria. 

OST’s budgetary resources have grown and the range of program 

responsibilities has changed. Budgetary resources for OST increased 

significantly, from $13.2 million in fiscal year 1995 to $204.2 million in 

fiscal year 2003 (in constant 2002 dollars), totaling over $1 billion.8

 This 

increase can be attributed to the introduction of new allocations and large 

increases for existing ones. The NIJ director decides how to allocate 

certain appropriated funds to the various NIJ components, including OST. 

About $749.7 million, or 72 percent, of OST’s total budgetary resources 

was either directed to specific recipients or projects by public law, subject 

to congressional committee report guidance designating specific 

recipients or projects, or directed from the reimbursements from other 

Justice and federal agencies in exchange for OST managing their projects. 

Corresponding with the designation of spending for specific recipients and 

projects, the range of OST’s programs changed, from primarily law 

enforcement and corrections to include broader public safety technology 

R&D, such as for improving school safety and combating terrorism. 

OST’s budgetary resources9

 include both funding received via Justice 

appropriations accounts as well as reimbursements from other Justice and 

federal agencies. First, OST receives funding via three appropriations 

accounts enacted in the appropriations law for the Justice Department. 

From these appropriations accounts, OJP allocates amounts to NIJ. The 

NIJ director suballocates part of the NIJ funds for OST programs. In 

addition, OST receives reimbursements from other Justice and federal 

agencies in exchange for OST’s management of specific projects of those 

agencies, such as ballistic imaging evaluation for the FBI. Table 1 lists NIJ 

 8

Figures do not include funding for management and administration expenses, salaries, and 

unobligated balances carried from one year to the next. 

9

For the purposes of this report, we will refer to both the funds OST receives via several 

Justice appropriations accounts as NIJ allocations as well as the reimbursements it 

receives as OST’s budgetary resources. 

OST’s Budgetary 

Resources Have 

Grown and Program 

Responsibilities Have 

Changed 

Budgetary Resources for 

OST’s Programs 
Page 7 GAO-04-198 Justice's Office of Science and Technology 

allocations from the Justice appropriations accounts that go toward 

funding OST programs. 

Table 1: Flow of Budgetary Resources to OST’s Programs 

Justice appropriation accounts NIJ’s allocations to OST programs 

Justice Assistance NIJ Base: NIJ uses base funds for research, development, demonstration, 

and dissemination activities. 

Counterterrorism R&D:

a

 NIJ sponsors research, development, and 

evaluations and tools to help criminal justice and public safety agencies deal 

with critical incidents, including terrorist acts. 

State and Local Law Enforcement Assistance 

(SLLEA) 

Local Law Enforcement Block Grant (LLEBG): NIJ allots its R&D portion 

of LLEBG funds to OST to assist local units of government to identify, select, 

develop, modernize, and purchase new technologies for law enforcement 

use. 

Community Oriented Policing Services 

(COPS) 

Crime Identification Technology Act (CITA): CITA activities include 

upgrading and integrating national, state, and local criminal justice record, 

identification systems, and funding multi-jurisdictional, multi-agency 

communications systems, and improving forensic science capabilities, 

including DNA analysis. 

Safe Schools Technology R&D: OST’s Safe Schools Technology R&D 

program uses three methods for improving school safety: needs 

assessments and development of technical partners, technology R&D, and 

technical assistance. 

Crime Lab Improvement Program (CLIP): CLIP activities include providing 

equipment, supplies, training, and technical assistance to state and local 

crime laboratories to establish or expand their capabilities and capacities to 

perform various types of forensic analyses. 

DNA Backlog Reduction: This seeks to eliminate public crime laboratories’ 

backlogs of DNA evidence as soon as possible. 

Paul Coverdell National Forensic Sciences Improvement Act (NFSIA): 

This provides funding to state and local laboratories to improve the quality, 

timeliness, and credibility of forensic science services for criminal justice 

purposes. 

Reimbursements of funds from other Justice Department and federal 

agencies’ accounts: Reimbursable activities have included ballistic imaging 

evaluation from the FBI, a study of communications interoperability (the 

ability to communicate across different public safety agencies and 

jurisdictions) requirements from the Defense Advanced Research Projects 

Agency, and death investigator guidelines from the Centers for Disease 

Control and Prevention. 

Source: GAO analysis of OST data. 

a

In fiscal year 1999, OST’s counterterrorism R&D programs received funding through the Justice 

Department’s Counterterrorism Fund appropriation account. 

OST’s budgetary resources almost quadrupled from fiscal year 1995 to 

1996, increased 70 percent from fiscal year 1999 to 2000, and increased 

63 percent from fiscal year 2001 to 2002. While resources decreased 
Page 8 GAO-04-198 Justice's Office of Science and Technology 

24 percent from fiscal year 2002 to 2003, OST’s fiscal year 2003 level still 

represents a 157 percent increase over the fiscal year 1999 level. 

Figure 1: OST’s Budgetary Resources in Constant 2002 Dollars, Fiscal Years 

1995–2003 

Notes: Figures do not include funding for management and administration expenses, such as 

salaries. 

The $103.4 million increase from fiscal year 2001 to 2002 is largely attributable to increases of 

$55.6 million in reimbursable agreements, $24.3 million in DNA Backlog Reduction allocation, and 

$15.4 million in the Crime Lab Improvement Program allocation. 

The sharp decrease in OST’s budgetary resources from fiscal years 2002 to 2003 is largely attributed 

to the elimination of counterterrorism R&D allocation (from $45.3 million in fiscal year 2002), which 

moved to the Department of Homeland Security, and a decrease of $26.2 million from reimbursable 

agreements. 

Our analysis of OST’s yearly budgetary resources from fiscal year 1995 to 

fiscal year 2003 showed that the overall increase can be attributed to the 

introduction of new NIJ allocations and large increases for existing ones. 

The NIJ allocations that contributed to the overall increase in OST’s 

budgetary resources are most notably the Crime Lab Improvement 

Program, DNA Backlog Reduction, Safe Schools Technology R&D, and 

Counterterrorism R&D allocations. Table 2 shows figures for all years in 

constant 2002 dollars. 

Certain Allocations 

Contributed to the 

Increase in Budgetary 

Resources since 1995 

Dollars in millions

0

25

50

75

100

125

150

175

200

225

250

275

1995 1996 1997 1998 1999 2000 2001 2002 2003

Fiscal year

Source: OST.
Page 9 GAO-04-198 Justice's Office of Science and Technology 

All dollar figures used in this narrative are in constant 2002 dollars, except 

as noted otherwise. 

Fiscal years 1995-1996: The $39.4 million (298 percent) increase from 

$13.2 million to $52.6 million primarily came from two NIJ allocations 

totaling $35.4 million. 

• Local Law Enforcement Block Grant (LLEBG) initiated with $22.2 million. 

• Reimbursement of funds increased by $13.2 million (471 percent) from 

$2.8 million to $16.0 million. 

Fiscal years 1999-2000: The $55.6 million (70 percent) increase from 

$79.5 million to $135.1 million primarily came from three NIJ allocations 

totaling $51.7 million. 

• DNA Backlog Reduction initiated with $15.6 million. 

• Safe Schools Technology R&D allocation initiated with $15.6 million.10

• Counterterrorism R&D increased by $20.5 million (193 percent) from $10.6 

million to $31.1 million. 

Fiscal years 2001-2002: The $103.4 million (63 percent) increase from 

$164.6 million to $268.0 million primarily came from three NIJ allocations 

totaling $95.3 million. 

• Reimbursement of funds increased by $55.6 million (209 percent) from 

$26.6 million to $82.2 million. 

• DNA Backlog Reduction increased by $24.3 million (227 percent) from 

$10.7 million to $35 million. 

• Crime Lab Improvement Program increased by $15.4 million (79 percent) 

from $19.6 million to $35 million. 

To be consistent with the report narrative and to show trends, figures in 

table 2 are in constant 2002 dollars. A table with the figures in current 

dollars can be found in appendix II. 

 10In fiscal year 1999, NIJ used the LLEBG allocation to meet congressional guidance to 

spend $10 million on a new Safe School Initiative. The following year NIJ’s Safe Schools 

Technology R&D funding was introduced with $15 million. The OST funding was not 

reduced as a result of the $15 million increase for the Safe Schools Technology R&D. 
Page 10 GAO-04-198 Justice's Office of Science and Technology 

Table 2: Budgetary Resources in Constant 2002 Dollars for OST’s Programs by NIJ Allocation, Fiscal Years 1995-2003 

Dollars in millions 

NIJ allocations for OST programs 1995 1996 1997 1998 1999 2000 2001 2002 2003 Totala

NIJ Base 10.4 13.3 12.7 14.8 20.3 19.1 29.0 27.1 32.3 179.1 

Local Law Enforcement Block Grant 

(LLEBG) 0 22.2 21.7 21.4 21.2 20.8 20.2 20.0 19.6 167.1 

Crime Identification Technology Act 

(CITA) 0 0 0 0 0 4.4 4.3 1.4 0 10.1 

Safe Schools Technology Research 

and Development 0 0 0 0 0 15.6 17.7 17.0 16.6 66.9 

Crime Lab Improvement Program 

(CLIP) 0 1.1 3.3 13.4 15.9 15.6 19.6 35.0 39.6 143.4 

DNA Backlog Reductionb 0 0 0 0 0 15.6 10.7 35.0 35.2 96.5 

Paul Coverdell National Forensic 

Sciences Improvement Act (NFSIA)b

0 0 0 0 0 0 0 5.0 4.9 9.9 

Counterterrorism R&D 0 0 10.9 12.9 10.6 31.1 36.5 45.3 0 147.2 

Reimbursements from other Justice 

and federal agencies 2.8 16.0 0 8.9 11.5 13.0 26.6 82.2 56.0 217.1 

Totala 13.2 52.6 48.6 71.4 79.5 135.1 164.6 268.0 204.2 1037.1 

Source: GAO analysis of OST data. 

a

Totals might not add due to rounding. 

b

In fiscal years 2000 and 2001, DNA Backlog Reduction was funded as DNA Combined DNA Index 

System (CODIS) Backlog Reduction. In fiscal years 2002 and 2003, both the DNA Backlog Reduction 

and Coverdell NFSIA allocations were funded within DNA CODIS Backlog Reduction. 

OST had a $63.8 million (24 percent) decrease in total budgetary resources 

from fiscal years 2002 to 2003, largely attributed to its not receiving fiscal 

year 2003 Counterterrorism R&D resources, which totaled $45.3 million in 

fiscal year 2002. According to OST, its counterterrorism resources were 

transferred to the Department of Homeland Security’s Office of Domestic 

Preparedness. There was also a $26.2 million decrease in the 

reimbursement of funds from other agencies. However, OST’s fiscal year 

2003 level still represents a 157 percent increase from fiscal year 1999. 

The range of OST’s program responsibilities has changed over the years 

from primarily law enforcement and corrections to include broader public 

safety technology R&D. This has happened as more and more of OST’s 

budgetary resources were directed to be spent on specific recipients and 

projects. Appropriated funds, for example, are sometimes designated for 

specific recipients or projects in public law. In addition, guidance on the 

spending of appropriated funds may be provided through congressional 

Range of OST’s Program 

Responsibilities Has 

Changed 
Page 11 GAO-04-198 Justice's Office of Science and Technology 

committee reports. Of the more than $1 billion (in constant 2002 dollars) 

that OST programs received from fiscal years 1995 to 2003, $532.6 million, 

or 51 percent, was designated for specific recipients and projects in public 

law or subject to guidance in committee reports designating specific 

recipients or projects.11 Of the $532.6 million, $249.8 million was 

designated in public law for specific recipients or projects while $282.8 

million was specified in committee report guidance for specific recipients 

or projects.12

In addition to the $532.6 million designated in public law for specific 

recipients or projects or subject to guidance in committee reports for 

specific recipients or projects, another $217.1 million was reimbursements 

from other Justice and federal agencies in exchange for OST’s 

management of specific projects of those agencies. Thus, the total 

spending either directed for specific recipients and projects through public 

law, subject to committee report guidance designating specific recipients 

or projects, or received as reimbursements, amounts to $749.7 million, or 

72 percent, of OST’s total budgetary resources. 

The range of OST’s program responsibilities has changed to include such 

areas as school safety and counterterrorism. In fiscal year 1999, a Safe 

Schools Initiative program was established pursuant to conference 

committee report guidance13 with $10 million14 directing NIJ to develop 

school safety technologies. In another example, OST’s counterterrorism 

R&D program, initially funded by public law in fiscal year 1997,15 received 

$147.3 million through fiscal year 2002, $96.6 million of which was 

specified in conference report guidance for three recipients from fiscal 

 11We separated reimbursements from this total because they included projects that were 

not originally allocated to OST, although those projects also may have been specified in 

public law and committee reports. 

12Included in the $249.8 million was $143.5 million for the CLIP project. Committee report 

guidance further designated $107.0 million of that $143.5 million for specific recipients. 

Given that we have included the $107.0 million in the amounts designated in public law for 

specific recipients or projects, we excluded it from the committee report guidance category 

to avoid double counting. 

13H.R. Conf. Rep. No. 105-825, at 1020-21 (1998). 

14For this effort, NIJ initially allocated Local Law Enforcement Block Grant funds to OST. 

15P.L. 104-208, 110 Stat. 3009, 3009-13 (1996). 
Page 12 GAO-04-198 Justice's Office of Science and Technology 

years 2000 to 200216—Oklahoma City National Memorial Institute for the 

Prevention of Terrorism ($37.8 million), Dartmouth College’s Institute for 

Security Technology Studies ($51.8 million), and the New York 

University’s Center for Catastrophe Preparedness and Response 

($7 million). 

OST’s program responsibilities have also changed to expand the focus on 

investigative and forensic sciences. Our review of OST’s budgetary 

resources for fiscal years 1995 through 2003 shows that budgetary 

resources for investigative and forensic sciences equals at least 

$342.1 million in constant fiscal year 2002 dollars,17 or about one-third, of 

its $1 billion in budgetary resources, as shown in table 3. The proportion of 

investigative and forensic sciences annual funding to total OST funding 

rose from 6 percent ($800,000) in fiscal year 1995 to 52 percent 

($106.0 million) in fiscal year 2003. 

 16H.R. Conf. Rep. No. 106-479, at 161 (1999); H.R. Conf. Rep. No. 106-1005, at 226 (2000); 

and H.R. Conf. Rep. No. 107-278, at 86-87 (2001). 

17The total amount of budgetary resources for investigative and forensic sciences is likely 

to be larger. However, because of the limitations in detail in the budget documents we 

received from OST, we could not determine the amount of funding for investigative and 

forensic sciences within certain NIJ Base and LLEBG projects, such as within OST’s 

technology center network and unspecified NIJ-directed projects. 
Page 13 GAO-04-198 Justice's Office of Science and Technology 

Table 3: Budgetary Resources in Constant 2002 Dollars for OST’s Investigative and Forensic Sciences by NIJ Allocation, 

Fiscal Years 1995-2003 

Dollars in millions 

NIJ allocation containing funds for 

investigative and forensic sciences 1995 1996 1997 1998 1999 2000 2001 2002 2003 Totala

NIJ Base 0.6 0.6 0.4 1.5 6.2 5.6 5.5 5.0 4.3 29.6 

LLEBG 0 0 0 0 0 1.1 0 0 0 1.1 

CITA 0 0 0 0 0 0.8 1.3 0 0 2.0 

Safe Schools Technology R&D 0 0 0 0 0 0 0 0 0 0 

CLIP 0 1.1 3.3 13.4 15.9 15.6 19.6 35.0 39.6 143.4 

DNA Backlog Reduction 0 0 0 0 0 15.6 10.7 35.0 35.2 96.5 

Coverdell NFSIA 0 0 0 0 0 0 0 5.0 4.9 9.9 

Counterterrorism R&D 0 0 0 0 0 0 0 0 0 0 

Reimbursement of funds from other agencies 0.2 8.9 0 0 0 1.6 1.1 25.4 22.0 59.1 

Totala 0.8 10.5 3.6 14.9 22.1 40.2 38.5 105.4 106.0 342.1 

Source: GAO analysis of OST data. 

a

Totals might not add due to rounding. 

OST delivers many products, which we categorized into three groups, and 

uses various methods to deliver them. These three groups are 

(1) information dissemination and technical assistance; (2) the 

application, evaluation, and demonstration of existing and new 

technologies for field users; and (3) technology R&D. According to OST, as 

of April 2003, it had delivered 945 products since its inception.18

Furthermore, OST identified an additional 500 products expected from 

ongoing awards. Figure 2 shows our distribution of OST’s delivered 

products by group. We recognize, as OST officials told us, that the groups 

overlap and there is not a clean division between them. For example, 

while reports are associated with information dissemination, they may 

also result from the technology R&D group. OST has reviewed our 

classification of products and agrees that it is generally accurate. Because 

classification of some products is based on a judgment call, the 

proportions of products in each group should be considered 

approximations. 

 18Because NIJ’s science and technology efforts predate OST’s establishment in fiscal year 

1995, some of the products listed as delivered have award years prior to 1995. The earliest 

listed is 1983. 

OST Delivers Three 

Groups of Products 

Through Various 

Methods 
Page 14 GAO-04-198 Justice's Office of Science and Technology 

The following examples, while not exhaustive, indicate the wide range of 

OST’s products. 

• Reports on topics such as analysis of DNA typing data, linguistic methods 

for determining document authorship, a pepper spray projectile and 

disperser, and gunshot residue detection and interpretation. 

• Prototypes of products including ground-penetrating radar, ballistics 

matching using 3-dimensional images of bullets and cartridge cases, and 

an optical recognition system to identify and track stolen vehicles. 

• Evaluations of technology including prison telemedicine networks, police 

vehicles, and protective gear. 

• Guides on topics such as electronic crime scene investigation, use of 

security technologies in schools, and antennas for radio communications. 

For a more detailed description of OST’s products and further examples, 

see appendix III. 

Figure 2: GAO’s Grouping of OST’s 945 Delivered Products, as of April 2003 

Notes: See appendix III, table 7 for examples of the products within each group. Proportions should 

be considered approximations because some products overlap categories. 

Information dissemination and technical assistance represents about 

63 percent of OST’s delivered products. OST provides information to its 

customers in a variety of ways. For example, OST provides guidance to 

R&D laboratories on the needs of public safety practitioners. To public 

safety practitioners, OST recommends certain public safety practices, 

tools, and technologies. Through its Office of Law Enforcement Standards, 

OST’s Range of Products 

Information Dissemination 

and Technical Assistance 

20%

63% 17%

Application, evaluation, and demonstration of 

existing and new technology for field users

(191) 

Technology R&D

(161)

Information dissemination and technical 

assistance

(593)

Source: GAO analysis of OST data. 
Page 15 GAO-04-198 Justice's Office of Science and Technology 

OST develops performance standards to ensure that commercially 

available public safety equipment, such as handheld and walk-through 

metal detectors, meets minimum performance requirements. OST also 

helps its customers enhance their technical capacities by providing them 

with training and technical assistance through its Crime Lab Improvement 

Program (which also provides supplies and equipment), DNA Backlog 

Reduction Program, and network of technology centers. OST also uses the 

R&D expertise and experience of already established laboratories and 

other R&D organizations to provide additional guidance for managing 

specialized technology projects. Further, OST helps its customers receive 

surplus federal equipment by acting as their liaison to the equipment 

transfer program of the Department of Defense. For example, equipment 

transferred ranges from armored vehicles to boots and uniforms. 

In addition, OST sponsors conferences, workshops, and forums that bring 

together its customers, technologists, and policymakers. For example, it 

sponsors the Mock Prison Riot, an annual event demonstrating emerging 

technologies in riot training scenarios held at the former West Virginia 

Penitentiary in Moundsville, West Virginia. This event brings together 

corrections officers and vendors for technology showcases and training 

exercises. Also, OST sponsors the Innovative Technologies for Community 

Corrections Annual Conference, among others. 

Another OST product group is the application, evaluation, and 

demonstration of new and existing technologies, which represents about 

20 percent of OST’s delivered products. Some of OST’s programs apply 

existing technology solutions in new ways to assist public safety 

operations. Examples of the application of new and existing technologies 

include developing methods for the collection and analysis of chemical 

trace evidence left from explosives and a handheld computer device 

provided to bomb technicians in order to access bomb data at the scene of 

incidents. In addition, OST tests commercially available products through 

NIJ-certified laboratories to determine whether they are in accordance 

with national performance standards. Examples of products evaluated 

against standards include body armor, handcuffs, and semiautomatic 

pistols. OST’s evaluations also include conducting field tests to compare 

different commercially available products of the same type to allow users 

to select the product that best suits their needs. OST also demonstrates 

technology resulting from R&D directly to its customers through OSTsponsored events. For example, the Critical Incident Response 

Technology Seminar, formerly known as the Operation America, 

demonstrates live-fire simulation for bomb technicians. The annual Mock 

Application, Evaluation, 

and Demonstration of New 

and Existing Technologies 
Page 16 GAO-04-198 Justice's Office of Science and Technology 

Prison Riot demonstrates emerging technologies for use by corrections 

officers and tactical team members. 

About 17 percent of OST’s delivered products were related to technology 

R&D, which involves the development of prototype devices, among other 

efforts.19 According to OST, R&D in its early stages includes development 

of prototypes and demonstration that a principle can be proven. Applied 

R&D, which also involves the development of prototypes, includes 

technologies that are made available to public safety agencies, generally 

through OST-assisted commercialization. Examples of products resulting 

from OST’s applied R&D range from a bomb threat training simulator, 

facial recognition technology for internet-based gang tracking, to a 

personal alarm and location monitoring system for corrections officers. 

According to OST, R&D in its early stages begins with testing technology 

concepts, exploring solutions, and deciding whether continued 

development is warranted. If OST decides to support product development 

and if it has available funds, it awards funding to develop, demonstrate, 

and evaluate an experimental prototype, which is then further developed 

into an initial engineering prototype, and then demonstrated and 

evaluated. If the prototype proves successful, OST demonstrates a “near 

commercial” model to its customers for their evaluation. 

While OST does not directly commercialize the results of its technology 

R&D, it does provide prototypes to local users for field-testing and assists 

in linking prototypes with potential commercial developers. OST officials 

believe it would be a conflict of interest and therefore inappropriate for 

them to promote one vendor or technology over another or try to dictate 

what equipment their customers should purchase. OST’s role in 

commercialization is to bring technologies and potential manufacturers 

together so that the manufacturers can determine the feasibility of 

commercializing the technologies. 

 19While some of the products resulting from technology R&D are similar to those of the 

application, demonstration, and evaluation of new and existing technologies group, the 

primary distinction is that the former includes the development of prototypes and the latter 

generally does not. 

Technology R&D 
Page 17 GAO-04-198 Justice's Office of Science and Technology 

OST delivers its products to its customers through a variety of methods. 

(We recognize that products are sometimes delivery methods. For 

example, a publication can be both a product resulting from research and 

a method of information dissemination.) Besides publications, OST’s 

methods for delivering information and technical assistance include mass 

mailings; downloadable material from its Web site; panels, boards, and 

working groups; training, support, and presentations; and programs to 

enhance the capacity of public safety agencies. 

OST also delivers its products related to application, evaluation, and 

demonstration through various means. For example, private industry 

provides new and existing technologies to OST; in turn, OST informs its 

customers of the results of using these technologies in new ways. OST 

publishes user guides and the test results of its evaluations of 

commercially available equipment (both standards-based and comparisonbased). Seeking to further educate its customers, OST demonstrates new 

technology at technology fairs, providing “hands on” opportunities to use 

it. 

For its R&D products, OST may test “near commercial” prototypes in 

particular settings. For example, OST may install in a police agency a 

prototype technology that facilitates communications among public safety 

agencies and across jurisdictions. If the technology is effective, the police 

agency may incorporate the technology directly into its operations, before 

the technology has become a commercial product. 

OST’s efforts to measure its performance results, including the usefulness 

and effectiveness of its products, do not fully meet applicable 

requirements. To help Justice comply with the Government Performance 

and Results Act of 1993 (GPRA), OST establishes goals and develops 

performance measures to track its progress. GPRA, which mandates 

performance measurements by federal agencies, requires, among other 

things, that each agency measure or assess relevant outputs and outcomes 

of each program activity.20 According to GPRA, the Office of Management 

and Budget (OMB), and GAO, outcomes assess actual results as compared 

with the intended results or consequences that occur from carrying out a 

 20Performance measures are to be included in the agency performance plan covering each 

program activity set forth in the budget of such agency. Program activity, in this case, 

refers to projects and activities that are listed in program and financing schedules of the 

annual Budget of the United States Government. 

OST’s Methods for 

Delivering its Products 

OST’s Performance 

Measurement Efforts 

Do Not Fully Meet 

Requirements 
Page 18 GAO-04-198 Justice's Office of Science and Technology 

program or activity. Outputs count the goods and services produced by a 

program or organization. Intermediate measures can be used to show 

progress to achieving intended results. Subsequent OMB and committee 

report guidance on GPRA, and previous GAO reports21 recognize that 

output measures can provide important information in managing 

programs. However, committee report guidance emphasizes using 

outcome measures to aid policy makers because such measures are key to 

assessing intended results. 

The performance measures that OST has developed do not measure 

results. According to the NIJ director, the Assistant Attorney General 

(AAG) in April 2002 issued a memorandum requiring NIJ, including OST, 

to develop outcome measures for fiscal year 2004. In August 2002, the NIJ 

Director responded by stating that OST had indeed developed outcome 

measures for its programs. In its fiscal year 2004 performance plan,22 OST 

established goals for 11 of its initiatives23 and developed 42 measures for 

assessing the achievement of those goals. However, based on our review 

of OST’s performance plan, OMB guidance on GPRA, and GAO definitions 

of outcome, output, and intermediate measures, we determined that of the 

42 measures, none were outcome-oriented, 28 were output-oriented, and 

14 were intermediate. See table 4 for GAO’s determination of the measures 

and appendix VI for further details of our results. 

 21U.S. General Accounting Office, Managing for Results: An Agenda to Improve the 

Usefulness of Agencies’ Annual Performance Plans, GAO/GGD/AIMD-98-228 (Washington, 

D.C.: Sept. 8, 1998). 

22Annual performance plans describe a department component’s goals and performance 

targets in support of the department’s long-term strategic goals and targets. In its fiscal 

year 2004 performance plan, OST reported actual performance data for fiscal year 2002, 

enacted plans for fiscal year 2003, and performance plans for fiscal year 2004. 

23Initiatives in this sense encompass portfolio areas, programs, and projects. 

OST Performance 

Measures Do Not Measure 

Results 
Page 19 GAO-04-198 Justice's Office of Science and Technology 

Table 4: GAO’s Assessment of the 42 Measures OST Developed for 11 of Its 

Initiatives 

Type of measure 

OST’s initiatives Output Intermediate Outcome 

1. Convicted Offender DNA Backlog Reduction 

Program 0 3 0 

2. No Suspect DNA Backlog Reduction Program 0 1 0 

3. Paul Coverdell National Forensic Sciences 

Improvement Grants Program 0 1 0 

4. Critical Incident Response Technology Initiative 4 1 0 

5. DNA Research & Development 4 0 0 

6. Law Enforcement Technology Research and 

Development 4 1 0 

7. School Safety Technology 3 0 0 

8. Crime Lab Improvement Program 4 6 0 

9. Office for Law Enforcement Standards 3 0 0 

10. Smart Gun 4 0 0 

11. OST’s network of regional centers (known as 

the National Law Enforcement and Corrections 

Technology Center system) 2 1 0 

Source: GAO analysis of OST data. 

According to Justice officials, R&D activities present measurement 

challenges because outcomes are difficult or costly to measure. As the NIJ 

Director pointed out, a May 2002, White House OMB and Office of Science 

and Technology Policy memorandum concluded that agencies should not 

have the same expectations for measuring the results of basic R&D as they 

do for applied R&D.24 According to NIJ, relatively little of OST’s work is 

basic R&D. As shown earlier, most of OST’s products are related to 

information dissemination and technical assistance and the application, 

evaluation, and demonstration of existing and new technologies for field 

users. 

 24According to the OMB document, Budget of the United States Government (Analytical 

Perspectives) for fiscal year 2004, basic R&D is defined as systematic study directed 

toward greater knowledge or understanding of fundamental aspects of phenomena and of 

observable facts without specific applications toward processes or products in mind. 

Applied R&D is defined as systematic study to gain knowledge or understanding necessary 

to determine the means by which a recognized and specific need may be met. 
Page 20 GAO-04-198 Justice's Office of Science and Technology 

We recognize that OST’s task in relation to measuring the results of even 

non-basic research is complex in part because of the wide array of 

activities it sponsors, and because of inherently difficult measurement 

challenges involved in assessing the types of programs it undertakes. For 

example, programs that are intended to deter crime face measurement 

issues in assessing the extent to which something (crime) does not 

happen. Nevertheless, improvement in measurement of program results is 

important to help OST ensure it is doing all that is possible to achieve its 

goals. It is worth noting that an outcome measure in relation to one OST 

program was discussed by the NIJ Director in a May 2002 statement to 

Congress. In this statement, the Director provided an example of an 

outcome from the Convicted Offender DNA Backlog Reduction Program. 

The Director stated that as a direct result of the program, approximately 

400,000 convicted offender samples and almost 11,000 cases with no 

suspect were analyzed. According to the NIJ Director, as of May 14, 2002, 

more than 900 “hits” had been made on the FBI’s Combined DNA Index 

System (CODIS) database as a direct result of the program, that is, 900 

cases previously unsolved had been reopened. This information indicates 

how the program is achieving its intended results in addressing unsolved 

cases. Although this example seems to be a credible outcome measure, it 

is not included in OST’s fiscal year 2004 performance plans. 

OST efforts to measure information dissemination effectiveness have been 

limited. One of the purposes of GPRA is to improve federal program 

effectiveness and public accountability by promoting a new focus on 

results, service quality, and customer satisfaction. Surveys to gauge 

customer satisfaction represent one step toward finding out whether 

customers have received information and whether they deem it of value. 

However, these surveys have limitations in determining the extent to 

which the information has been acted upon and resulted in intended 

improvements. Thus, surveys such as these are more likely to be 

intermediate measures (Did information get transferred?) than outcome 

measures (Did information get transferred, acted upon, and achieve a 

result?). 

In 1998, NIJ initiated an effort to report the results of surveys to measure 

the satisfaction of participants at all conferences, workshops, and seminar 

series.25 OST reported on the “grantee level of satisfaction with NIJ 

 25The surveys were done to determine if participants were satisfied with the conference as 

a vehicle of information dissemination. 

Limitations in OST’s 

Efforts to Measure 

Effectiveness of 

Information Dissemination 
Page 21 GAO-04-198 Justice's Office of Science and Technology 

conferences” for fiscal years 1998-2000. However, in the fiscal years 2001-

2004 GPRA performance plans, OST discontinued tracking the surveys 

because OJP and NIJ had ceased tracking these data as a performance 

measure. 

In fiscal year 2001, OST attempted to evaluate the effectiveness and value 

of its TECHbeat newsletter. The survey sample of 5,500 was taken from a 

distribution of major readership groups on TECHbeat’s mailing list of 

20,674. According to OST, the response rate for the survey was too low to 

produce statistically valid results: only 124 completed or substantially 

completed responses were collected. The surveyors also experienced a 

very low return on follow-up phone queries. According to the study, the 

primary reason for the exceedingly low response rate was that so many 

individuals on the mailing list had either changed jobs or were completely 

unfamiliar with TECHbeat. Given these results, OST is trying to improve 

the management and distribution of TECHbeat.26

In fiscal year 2001, OST attempted to launch another effort to measure 

program results, service quality, and customer satisfaction, but funding for 

the effort was not provided. OST requested funding for an evaluation to 

measure the success of its outreach efforts, including those by its 

technology centers. The evaluation was to determine customer 

satisfaction with its strategies for outreach and communication and with 

its products. Specifically, OST planned to measure user satisfaction of the 

content, format, and delivery mechanisms of its efforts, such as 

technology information and assistance. 

 26To address issues with the mailing lists, the technology centers have shipped a larger 

portion of copies to agencies, in bulk, and to individuals who have actively requested 

copies and supplied their addresses; continued to purchase the most current version of the 

National Directory of Law Enforcement Administrators, Correctional Institutions and 

Related Agencies to update their mailing list; and modified mailing labels to include the 

addressee and “...or Training Officer” in case the addressee is no longer with that agency. 
Page 22 GAO-04-198 Justice's Office of Science and Technology 

In fiscal years 1998 and 1999, OST funded eight outside studies of some of 

its science and technology initiatives (see table 5).27 Our review of these 

studies showed that seven of the eight studies focused on management 

and organizational processes, and one was outcome-oriented.28

Management and process evaluations can be useful tools for examining 

how a program is operating and can offer insights into best practices. They 

do not assess whether a program is achieving its intended results. 

 27Initiatives in this sense encompass portfolio areas, programs, and projects. 

28GPRA establishes two approaches for assessing an agency’s performance: annual 

measurement of program performance against goals outlined in a performance plan and 

program evaluations to be conducted by the agency as needed. Evaluations can play a 

critical role in helping to address measurement and analysis challenges. Performance 

measurement is the ongoing monitoring and reporting of program accomplishments, 

particularly progress toward established goals. Program evaluations are individual 

systematic studies conducted periodically or on an ad hoc basis to assess how well a 

program is working. See U.S. General Accounting Office, Performance Measurement and 

Evaluation: Definitions and Relationships, GAO/GGD-98-26 (Washington, D.C.: April 

1998). 

Most Studies of Other OST 

Initiatives Have Focused 

Primarily on Process 
Page 23 GAO-04-198 Justice's Office of Science and Technology 

Table 5: OST’s Outside Studies of Its Initiatives 

Outside study topics Focus of study Type of study Date completed 

1. National Law Enforcement and Corrections 

Technology Centers (NLECTC) Programa

Management, oversight, structure, 

organization, and operations 

Process October 1998 

2. Counterterrorism Technology Portfolio Organization, funding, program process Process June 1999 

3. Investigative and Forensic Sciences 

Technology Portfolio 

Program and structure, management, 

policies, procedures, lines of control, and 

funding 

Process August 1999 

4. Less-Than-Lethal Technology Portfolio Management, processes, and organization Process September 1999 

5. Southwest Border States Antidrug Information 

System 

Program efficacy, including awareness of 

the program, and its value and usefulness 

or benefits to customers 

Outcome October 1999 

6. Law Enforcement and Corrections Technology 

Advisory Council Priorities and Technology 

Portfolio Interaction 

Management and coordination, processes, 

organizational challenges 

Process February 2000 

7. Critical Incident Response and Management 

Crime fighting Technology Program for State 

and Local First Responder Teams 

Options for planning, organization, mission, 

management, budget, and 

recommendations 

Process September 2000 

8. Standards Initiative Recommendations for the planning, 

organization, and management of the 

proposed initiative expected to be a part of 

#7 above 

Process September 2000 

Source: OST. 

a

In this report we refer to the National Law Enforcement and Corrections Technology Centers as 

OST’s network of technology centers. 

The Homeland Security Act of 2002 requires NIJ29 to transmit to Congress 

by late November 2003 a report assessing the effectiveness of OST’s 

existing system of technology centers and to identify the number of 

centers necessary to meet the technology needs of federal, state, and local 

law enforcement in the United States. According to NIJ, in response to the 

Homeland Security Act requirement, it has initiated a study to assess the 

impact and effectiveness of the technology center system and how it can 

be enhanced to meet the evolving science and technology research and 

technology needs of the state and local public safety community. NIJ also 

stated that the report would address the functions that the technology 

center system must provide to transfer NIJ’s research and development 

results to practice in the criminal justice system. NIJ and OST have failed 

 29The Homeland Security Act actually directs the “Director” of OST to transmit the report. 

After reorganizing in early 2003, NIJ now calls this position the assistant NIJ director for 

science and technology. 

Efforts Are Under Way to 

Address Performance 

Measurement of 

Technology Centers 
Page 24 GAO-04-198 Justice's Office of Science and Technology 

to provide us with information detailing the methodology of the study, so 

we cannot comment on the likelihood that this study will produce the 

information sought by Congress. Additionally, according to OJP, the 

technology centers are in the process of developing outcome measures to 

demonstrate the impact of their activities. 

According to NIJ, OJP has implemented additional performance measures 

developed in May 2003 that will apply to NIJ, including OST. However, OJP 

said it would defer implementing the measures related to the technology 

centers until the results of the technology center study are known and NIJ 

has a chance to take action, if warranted. 

We acknowledge that measuring results using outcome measures is 

difficult, and may be especially so in relation to some of the types of 

activities undertaken by OST. Indeed, given the types and wide range of 

program goals for OST efforts—solving old crimes, saving lives, and 

reducing property loss—it may be the case that for some programs 

intermediate measures represent the best feasible measure of results. We 

note that approximately 63 percent of OST’s products fall into the category 

of information dissemination and technical assistance, aimed at informing 

customers and ultimately encouraging adoption of research results that 

lead to increased efficiency and effectiveness. There are strategies 

available that have been used by other federal agencies to take steps 

toward assessing the effectiveness of information dissemination and 

technical assistance efforts. For example, a recent GAO report30 outlines 

various strategies to assess media campaigns and informational seminars, 

including immediate post workshop surveys and follow-up surveys and the 

use of logic models to define measures of a program’s progress toward 

intended results and long-term goals. 

Given the wide range of its products, OST has the potential to significantly 

improve the technological capabilities of federal, state, and local public 

safety agencies. However, the lack of information about the results of 

program efforts, or the assessment of progress toward goals, means that 

little is known about their effectiveness. While developing outcome 

 30U.S. General Accounting Office, Program Evaluation: Strategies for Assessing How 

Information Dissemination Contributes to Agency Goals, GAO-02-923 (Washington, D.C.: 

Sept. 30, 2002) 

Measuring Results Is 

Difficult but Feasible 

Conclusions 
Page 25 GAO-04-198 Justice's Office of Science and Technology 

measurements in many research-related programs is extremely difficult, 

there are various performance measurement strategies that other federal 

programs have used for assessing information dissemination, technical 

assistance and other R&D activities that might be applied to OST’s 

programs. It is important to develop outcome measurements where 

feasible, or intermediate measurements where appropriate, to assist 

Congress, OST and NIJ management, and OST’s customers to better assess 

whether investment in OST’s programs is paying off with improved law 

enforcement and public safety technology. 

To help ensure that OST does all that is possible to measure its progress in 

achieving goals through outcome-oriented measures, we recommend that 

the Attorney General instruct the Director of NIJ to reassess the measures 

OST uses to evaluate its progress toward achieving its goals and to better 

focus on outcome measures to assess results where possible. In those 

cases where measuring outcome is, after careful consideration, deemed 

infeasible, we recommend developing appropriate intermediate measures 

that will help to discern program effectiveness. 

We provided a copy of a draft of this report to the Attorney General of the 

United States for review and comment. In an October 30, 2003, letter, the 

Assistant Attorney General (AAG) for OJP commented on the draft. Her 

comments are summarized below and presented in their entirety in 

appendix VII. OJP also provided technical comments, which have been 

incorporated into this report where appropriate. 

In the AAG’s written response, the Justice Department concurred with our 

recommendation that NIJ reassess the measures OST uses to assess 

program outcomes. In response to our recommendation, the AAG reported 

that she has directed the NIJ Director, to reassess NIJ’s performance 

measures for OST and to refine them, where possible, in order to focus 

them more toward measuring outcomes. 

While the AAG agreed with our recommendation, she also made several 

other comments. First, she commented that developing numerical 

outcome measures like those urged by GAO is a particular challenge for 

R&D activities. As stated in our report, we recognize that measuring 

results using outcome measures is difficult and may be especially so in 

relation to some of the types of activities undertaken by OST. Our 

reference to a numerical measure is meant only as an example of how one 

of OST’s program activities can be linked to intended results. We believe 

Recommendation 

Agency Comments 

and Our Evaluation 
Page 26 GAO-04-198 Justice's Office of Science and Technology 

that further consideration of measures, both quantitative and qualitative, 

could improve the assessment of results for R&D as well as other OST 

programs. Our report also notes that relatively little of OST’s work is R&D. 

The majority of OST’s products are in the category of information 

dissemination and technical assistance. 

Second, the AAG noted that GAO did not reach any conclusions in its 

discussion of OST’s growth in budgetary resources, changes in program 

responsibilities, management of programs, and delivery of its products. 

The AAG noted that Justice believed that OST’s record is outstanding. 

Neither OST nor we can determine whether OST’s efforts in these areas 

are successful or otherwise, given that OST has not developed measures to 

assess their outcomes. Therefore, it is not possible to draw conclusions. 

Third, the AAG indicated that GAO did not discuss in detail that over onehalf of OST’s funds were designated by Congress for specific recipients 

and projects. She noted that GAO missed an opportunity to inform the 

requester of the impact of Congress’ recent decisions regarding OST. We 

reported that 51 percent of OST’s budgetary resources were designated for 

specific recipients and projects in public law or subject to guidance in 

committee reports. 

As agreed with your office, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 10 days from its 

issue date. At that time, we will send copies of the report to the Attorney 

General, appropriate congressional committees and other interested 

parties. We will also make copies available to others upon request. In 

addition, the report will be available at no charge on GAO’s Web site at 

http://www.gao.gov. Major contributors to this report are listed in 

appendix VIII. 
Page 27 GAO-04-198 Justice's Office of Science and Technology 

If you or your staff have any questions concerning this report, contact me 

on (202) 512-8777. 

Sincerely yours, 

Laurie E. Ekstrand 

Director, Homeland Security and Justice Issues 
Appendix I: Scope and Methodology 

Page 28 GAO-04-198 Justice's Office of Science and Technology 

To answer our objectives, we interviewed National Institute of Justice 

(NIJ) and Office of Science and Technology (OST) officials and collected 

documents at OST’s office in Washington, D.C., and at three of OST’s 

technology centers—the National center in Rockville, Maryland; West 

center in El Segundo, California; and Border Research and Technology 

Center in San Diego, California. We selected the Rockville center because 

of its proximity to Washington, D.C., and the other two centers because of 

their locations and particular areas of technology and technical 

concentrations. We also interviewed a small group of OST’s customers—

federal, state, and local law enforcement, and corrections and public 

safety officials—who were selected by officials at the El Segundo and San 

Diego centers. In addition, we analyzed information that is available on the 

National Institute of Justice’s public Web site. 

To determine OST’s budgetary resources and amounts from fiscal year 

1995 to fiscal year 2003 and the changes in OST program responsibilities, 

we reviewed NIJ and OST budget documents, interviewed officials in 

OST’s Technology Management and the OJP’s Office of Budget and 

Management Services, and reviewed pertinent appropriations laws and 

committee reports covering that period. To determine the amount of OST 

budgetary resources that were directed to specific recipients and projects, 

we compared OST’s budget documents that listed individual recipients and 

projects with the public laws and reports. We defined directed spending as 

spending for specific recipients and projects designated in appropriations 

laws or subject to congressional committee report guidance designating 

specific recipients or projects. We did not determine the amount of 

reimbursable projects designated in public laws or specified in committee 

reports because those projects were not originally allocated to OST. 

Instead, we considered all the reimbursable projects to be specific 

projects for which OST was directed pursuant to its agreements with other 

agencies on spending its reimbursable funds. 

To determine the changes in OST’s program responsibilities, we analyzed 

the year-to-year changes in its budget and program scope. To determine 

the amount of OST’s budgetary resources used for investigative and 

forensic sciences for fiscal years 1995-2003, we compared OST’s portfolio 

description and NIJ’s definition of forensic sciences with the individual 

budget program and project items listed in OST’s budget documents for 

each fiscal year. However, while we recognize that OST’s technology 

centers and their technical partners include investigative and forensic 

sciences in their provision of technical assistance, we did not attempt to 

determine the amount of center funds associated with investigative and 

forensic sciences because the budget documents we received from OST 

Appendix I: Scope and Methodology 
Appendix I: Scope and Methodology 

Page 29 GAO-04-198 Justice's Office of Science and Technology 

did not break out such amounts within the funding awarded to the centers. 

Therefore, our determination that $342.1 million of OST’s total funding 

supported investigative and forensic sciences did not include such 

amounts. 

To determine the amounts of funding awarded to the technology centers, 

we analyzed databases on all of the products OST has produced through 

April 2003 and the associated grants, interagency agreements, and 

cooperative agreements and their amounts. 

To determine the composition of OST’s products and how OST delivers 

the products to its customers, we analyzed OST documents and a database 

of all the products associated with its past and ongoing awards, from 

inception through April 2003, that were delivered or anticipated to be 

delivered. While the database included the award amounts associated with 

the products, it was not possible to reliably associate the award amounts 

for each product or type of product because multiple types of products 

could result from individual awards. We also conducted interviews with 

the parties mentioned above. 

For the budget and product data that OST provided us, we assessed the 

reliability of these data by examining relevant fields for missing data, 

conducting range checks to identify any potentially erroneous outliers and 

inspecting a subset of selected data elements that were common to two or 

more data sets. In addition, we independently verified selected budget 

data back to appropriations legislation and Committee reports. In 

conducting our analyses, we identified some potential data errors or 

reliability problems. When this occurred, we contacted agency officials to 

address and resolve these matters. However, we did not verify the budget 

or product data back to source materials. Overall, we determined that 

budget or product data provided to us is adequate for the descriptive 

purposes it is used in this report. 

We examined OST’s efforts to measure performance by interviewing 

officials on this matter at OJP, NIJ, and OST in the Washington, D.C., 

office along with officials and staff at the technology centers, and current 

and previous Advisory Council officials. We also reviewed related agency 

documents, such as the OJP mission statement and performance plans; 

NIJ strategic planning documents and website pages, annual performance 

plans and performance reports, and GPRA documents; policies and 

procedures, Department of Justice memoranda, OST internal planning and 

reporting documents, program descriptions and documentation, and other 

related documents. 
Appendix I: Scope and Methodology 

Page 30 GAO-04-198 Justice's Office of Science and Technology 

As part of our examination, we reviewed OST’s fiscal year 1997 to 2004 

goals and measures as presented in OST’s GPRA performance plans.1

 We 

focused our review on OST’s fiscal year 2004 performance plan and 

measures. As part of our review of these goals and measures, we made a 

determination as to whether the performance measure was output, 

outcome, or intermediate-oriented. To make this determination about the 

types of performance measures contained in OST’s performance plans, we 

compared the measures used in the plans with the requirements of GPRA, 

accompanying committee report, OMB’s guidance on performance 

measurement challenges (Circular A-11), Justice’s guidance to its 

components for preparing performance measures, and previous GAO work 

on GPRA.2

Also included in our examination of OST performance measurement 

efforts were studies prepared by external parties under grants from OST 

that reviewed selected OST initiatives such as portfolio areas, projects, 

and programs. In response to our request for all of OST’s efforts to assess 

its programs, OST provided eight outside studies funded from fiscal years 

1998 to 1999. For example, the Pymatuning Group, Inc., conducted an 

“Assessment of the National Law Enforcement and Corrections 

Technology Center (NLECTC) Program,” which described the operations 

of the OST’s regional technology centers network. We reviewed all eight of 

the outside studies for performance information on the OST initiatives 

being examined in the report. We examined the studies to determine 

whether they provided information that would be considered consistent 

with an outcome-oriented evaluation as defined by our criteria.3

The scope of this review was limited to OST, and therefore we cannot 

compare OST’s efforts to measure the performance of its programs or the 

amount of funding directed to specific recipients and projects with the 

efforts and funding of any other federal R&D agencies. We performed our 

 1

To determine the goal for each OST program included in the plan, we used the stated 

public benefit statement provided in the plan, except for the Law Enforcement Technology 

R&D program. 

2

See U.S. General Accounting Office, Agency Performance Plans: Examples of Practices 

That Can Improve Usefulness to Decisionmakers, GAO/GGD/AIMD-99-69 (Washington, 

D.C.: Feb. 26, 1999) for our guidance concerning intermediate-oriented measures and 

Managing for Results: Critical Issues for Improving Federal Agencies’ Strategic Plans, 

GAO/GGD-97-180 (Washington D.C.: Sept. 16, 1997). 

3

See U.S. General Accounting Office, Performance Measurement and Evaluation: 

Definitions and Relationships, GAO/GGD-98-26 (Washington, D.C.: April 1998). 
Appendix I: Scope and Methodology 

Page 31 GAO-04-198 Justice's Office of Science and Technology 

audit work from September 2002 to September 2003 in Washington, D.C., 

and other cited locations in accordance with generally accepted 

government auditing standards. 
Appendix II: Bugetary Resources for OST’s 

Programs in Current Year Dollars 

Page 32 GAO-04-198 Justice's Office of Science and Technology 

Table 6: Budgetary Resources in Current Dollars for OST’s Programs by NIJ Allocation, Fiscal Years 1995-2003 

Dollars in millions 

NIJ allocations for OST programs 1995 1996 1997 1998 1999 2000 2001 2002 2003 Totala

NIJ Base 9.2 12.0 11.7 13.8 19.2 18.4 28.6 27.1 32.8 172.9 

Local Law Enforcement Block Grant 

(LLEBG) 0 20.0 20.0 20.0 20.0 20.0 20.0 20.0 19.9 159.8 

Crime Identification Technology Act (CITA) 0 0 0 0 0 4.2 4.2 1.4 0 9.9 

Safe Schools Technology Research and 

Development 0 0 0 0 0 15.0 17.5 17.0 16.9 66.4 

Crime Lab Improvement Program (CLIP) 0 1.0 3.0 12.5 15.0 15.0 19.4 35.0 40.3 141.1 

DNA Backlog Reductionb 0 0 0 0 0 15.0 10.6 35.0 35.8 96.3 

Paul Coverdell National Forensic Sciences 

Improvement Act (NFSIA)b

0 0 0 0 0 0 0 5.0 5.0 10.0 

Counterterrorism R&D 0 0 10.0 12.0 10.0 30.0 36.0 45.3 0 143.3 

Reimbursements from other Justice and 

federal agencies 2.5 14.5 0 8.3 10.9 12.5 26.3 82.2 56.9 214.1 

Totala 11.7 47.5 44.7 66.6 75.1 130.2 162.6 268.0 207.6 1,013.8 

Source: GAO analysis of OST data. 

a

Totals might not add due to rounding 

b

In fiscal years 2000 and 2001, DNA Backlog Reduction allocations was funded as DNA CODIS 

Backlog Reduction. In fiscal years 2002 and 2003, both the DNA Backlog Reduction and Coverdell 

NFSIA allocations were funded within DNA CODIS Backlog Reduction.. 

Appendix II: Bugetary Resources for OST’s 

Programs in Current Year Dollars 
Appendix III: OST’s 10 Categories of Products 

Page 33 GAO-04-198 Justice's Office of Science and Technology 

While we divided OST’s products into three groups for our reporting 

purposes, OST divides them into 10 categories. (See table 7 for GAO’s 

3 groupings of OST’s 10 categories.) In regrouping OST’s 10 categories, we 

recognized, as OST officials told us, that the 10 categories overlap and 

there is not a clean division between them. We also recognized that many 

of OST’s products could also be considered a delivery method. For 

example, publications, such as the TECHbeat newsletter, are OST 

products that can also represent a method of delivery for OST technology 

information. OST has reviewed our classification of products and agrees 

that it is generally accurate. 

Table 7: GAO’s Groupings of OST’s Categories of Products and Examples of Each Category 

GAO’s 3 groupings of 

OST’s 10 categories OST’s 10 categories Examples of products 

1. Technology R&D 1. Results of the early stages of 

technology R&D include the 

development of prototypes and 

demonstration that a principle or 

concept can be proven. 

Results of investigating forensic techniques, studying potential 

less-than-lethal incapacitation technologies, and researching 

advanced weapons detection. 

 2. New applied technologies made 

available to public safety 

agencies, generally through 

commercialization. 

Improved bomb robots and electromagnetic concealed weapons 

detection. 

2. Application, evaluation, 

and demonstration of 

new and existing 

technologies for field 

users 

3. Existing technologies applied to 

new situations. 

Communications interoperability (the ability to communicate across 

different public safety agencies and jurisdictions), handheld 

computer devices for bomb investigators, and software tools to 

measure levels of school safety. 

 4. Product evaluations based on 

voluntary national performance 

standards and comparisons with 

like products. 

Ballistic and stab-resistant body armor, handcuffs, semi-automatic 

pistols, walk-through metal detectors; patrol vehicles, patrol vehicle 

tires, and replacement brake pads; cut-, puncture-, and pathogenresistant protective gloves. 

 5. Technology demonstrations. Annual Mock Prison Riot meeting demonstrates emerging 

technologies for use in hands-on riot training scenarios, and the 

annual Critical Incident Response Technology seminar (formerly 

called Operation America), in which bomb technicians practice livefire simulations. 

3. Information 

dissemination and 

technical assistance 

6. Information and guidance for 

public safety practitioners and 

those in R&D. 

Needs assessments of what public safety practitioners require, 

such as for combating electronic crime and terrorism; funding 

requirements for forensic science; investigative, selection, and 

application guides; and technology and training for small agencies. 

 7. Standards to ensure that 

commercially available public 

safety equipment meets 

minimum performance. 

Ballistic resistance of personal body armor and handheld and 

walk-through metal detectors. 

Appendix III: OST’s 10 Categories of Products 
Appendix III: OST’s 10 Categories of Products 

Page 34 GAO-04-198 Justice's Office of Science and Technology 

GAO’s 3 groupings of 

OST’s 10 categories OST’s 10 categories Examples of products 

 8. Enhanced capacity that gives 

agencies access to technologies 

and tools they otherwise might 

not have had funding for or 

access to. 

Technology assistance provided to OST’s customers by its 

regional centers; Crime Lab Improvement Program for establishing 

or expanding laboratories’ capacities for forensic analysis; the DNA 

Backlog Reduction Program for helping to eliminate DNA backlog, 

leading to the resolution of unsolved crimes. 

 9. Conferences, forums, and 

workshops that bring together 

practioners, technologists, and 

policymakers to form 

partnerships, communicate 

needs, and educate participants. 

Technical working groups of experienced practitioners and 

researchers working to improve investigation techniques and issue 

procedural guides. Panels and councils of public safety leaders, 

experts, and policymakers assisting OST and its regional centers 

in setting development priorities, launching technologies, 

identifying equipment problems, and enhancing understanding of 

technological issues and advances. Commercialization planning 

workshops involving developers and entrepreneurs interested in 

commercializing public safety technologies. 

 10. Technical expertise and 

oversight of technology projects 

provide additional oversight and 

guidance. 

Space and Naval Warfare Systems Command providing oversight, 

contracting, and administrative support for the NIJ User Centric 

Information Technology Program and Critical Incident 

Management System Testbed; the U.S Air Force Research 

Laboratory providing oversight and administrative support to the 

NIJ Concealed Weapons Detection and Personnel Location 

Technology Programs and hosting the NIJ-sponsored National 

Cyberscience Laboratory. 

Source: GAO analysis of OST data. 
Appendix IV: OST’s Portfolio Areas 

Page 35 GAO-04-198 Justice's Office of Science and Technology 

OST has organized its individual projects to develop, improve, and 

implement technology for public safety agencies into nine portfolio areas. 

As of April 2003, these portfolio areas included 

• critical incident technology, for first responders and investigators 

protecting the public in the event of critical incidents such as natural 

disasters, industrial accidents, or terrorist acts; 

• communications interoperability1 and information sharing, 

enhancing communication among public safety agencies through wired 

links, wireless radios, and information networks, even when disparate 

systems are involved; 

• electronic crime, supporting computer forensic laboratories, publishing 

guides for handling electronic evidence, and developing computer forensic 

tools; 

• investigative and forensic sciences, funding at the state and local levels 

for DNA-typing of convicted offenders and use of DNA-typing in the 

investigation of unsolved cases, and developing tools for forensic 

casework; 

• crime prevention technologies, including contraband detectors, sensors 

and surveillance systems, and biometric technologies; 

• protective systems technologies, including body armor; “smart” 

handguns, which fire only upon recognition of, for example, a certain 

handprint or password; puncture resistant gloves; better handcuffs; better 

concealed weapon detection; and personnel tracking and location 

technologies; 

• less-than-lethal technologies, developing alternatives to lethal force, 

including technologies involving electrical or chemical effects, light 

barriers, vehicle stopping, and blunt trauma, and evaluating and modeling 

the effects of these technologies; 

• learning technologies, developing technology tools for agencies to use in 

training their personnel, including use of the internet, CD-ROMs, and 

video-based and interactive simulations; and 

 1

Interoperability of communications is the ability to communicate across different public 

safety agencies and jurisdictions. 

Appendix IV: OST’s Portfolio Areas 
Appendix IV: OST’s Portfolio Areas 

Page 36 GAO-04-198 Justice's Office of Science and Technology 

• standards and testing, ensuring that the equipment public safety 

agencies buy is safe, dependable, and effective. 
Appendix V: OST’s Operations 

Page 37 GAO-04-198 Justice's Office of Science and Technology 

As with other federal agencies, OST’s operations involve multiple levels of 

internal organization and multiple kinds of external partners. OST’s 

multiple levels of organization include a Washington, D.C., office that 

manages its technology programs and a network of technology centers 

around the country that provide technical assistance to OST’s regional 

customers. OST also collaborates with other R&D entities, such as those in 

the Departments of Defense and Energy and public and private 

laboratories, by forming technical partnerships in order to leverage 

already established technical expertise and resources to support their 

program efforts. Another aspect of OST’s complex operations is the need 

to determine OST’s own priorities and the priorities of its customers, 

which involves Washington and regional center staff collaborating 

formally and informally with a myriad of federal, state, and local officials, 

as well as with one another. 

OST’s multiple levels of organization include a Washington, D.C., office 

and technology centers, as well as technical partnerships with 

government, public and private R&D and public safety organizations. As of 

September 2003, OST’s Washington office consisted of 25 full-timeequivalent Justice staff divided into three divisions and under the Assistant 

NIJ Director for OST.1

 Responsibility for managing these programs is 

divided among the three divisions. (See figure 3 for OST’s organizational 

structure.) 

 1

In addition, there were 2 federal detailees, 2 visiting scientists, and 32 on-site contractors 

supporting OST. 

Appendix V: OST’s Operations 

OST Has Multiple Levels of 

Organization 
Appendix V: OST’s Operations 

Page 38 GAO-04-198 Justice's Office of Science and Technology 

Figure 3: OST’s Organizational Structure 

• Research and Technology Development Division manages electronic 

crime (including cybercrime), critical incidents and counterterrorism, 

communications interoperability and information sharing, crime 

prevention, learning technology tools, less-than-lethal technologies, 

standards development, school safety, and corrections technologies. 

• Investigative and Forensic Sciences Division manages DNA-related 

R&D and other investigative and forensic sciences, such as fingerprint 

analysis, and includes the Crime Laboratory Improvement Program 

projects, DNA Backlog Reduction projects, and DNA research and 

development projects. 

• Technology Assistance Division, through the technology center 

network, provides training and technical advice to, and identifies 

technologies for, OST’s customers, and oversees OST’s network of 10 

technology centers (see figure 4). The technology centers are another 

source of technical advice for OST’s customers. 
Appendix V: OST’s Operations 

Page 39 GAO-04-198 Justice's Office of Science and Technology 

Figure 4: OST’s 10 Technology Centers and the Regions They Serve 

OST’s network of 10 technology centers provides technical assistance, 

among other things, to OST’s customers. From fiscal year 1995 to fiscal 

year 2003 (as of July 2003), funding support for the centers totaled 

$171.7 million. (See table 8 for funding by center.) The technology centers 

comprise six regional centers and four specialty centers. While the 

regional centers assist OST’s customers by region—Northwest, West, 

Rocky Mountain, Northeast, Southeast, and National—they are expected 

to coordinate and collaborate among one another regardless of where the 

resources and capabilities are located. Each of these 6 centers works with 

a regional advisory council comprising state and local law enforcement, 

corrections, and public safety representatives. 

As described below, the four specialty centers provide specialized 

expertise and services. 

• The Office of Law Enforcement Standards tests commercially available 

equipment and develops minimum performance standards for such 

equipment. 

OST’s Technology Centers 

Source: National Institute of Justice, Department of Justice.

Northwest

Anchorage, Alaska

Rocky Mountain

Denver, Colo.

West

El Segundo, Calif.

Border Research and

Technology Center

San Diego, Calif.

Northeast

Rome, N.Y.

Office of Law

Enforcement Standards

Gaithersburg, Md.

National

Rockville, Md.

Office of Law Enforcement

Technology Commercialization

Wheeling, W.Va.

Rural Law Enforcement

Technology Center

Hazard, Ky. 

Southeast

Charleston, S.C.
Appendix V: OST’s Operations 

Page 40 GAO-04-198 Justice's Office of Science and Technology 

• The Office of Law Enforcement Technology Commercialization, Inc., 

assists inventors and developers, among others, in commercializing 

technologies. 

• The Border Research and Technology Center aids in the development of 

technologies for agencies concerned with law enforcement at the northern 

and southern borders. 

• The Rural Law Enforcement Technology Center aids rural and smallcommunity law enforcement and corrections agencies. 

Table 8: Total Funds Awarded for the Operations, Maintenance, and Technical 

Support of OST’s 10 Technology Centers, Fiscal Years 1995-2003 

Dollars in millions 

Regional centers Total funding 

National, Rockville, Md. 20.4 

Northeast, Rome, N.Y. 11.7 

Southeast, North Charleston, S.C. 23.5 

Northwest, Anchorage, Alaska 2.8 

Rocky Mountain, Denver, Colo. 16.2 

West, El Segundo, Calif. 12.7 

Specialty centers 

Border Research Technology Center, San Diego, Calif. 8.2 

Office of Law Enforcement Standards, Gaithersburg, Md. 53.6 

Office of Law Enforcement Technology Commercialization, Wheeling, 

W.Va. 19.6 

Rural Law Enforcement Technology Center, Hazard, Ky. 3.0 

Total funding $171.7 

Source: OST. 

Notes: Figures are based on the current year values of each award. According to OST documents, 

the first award year for the Office of Law Enforcement Standards in support of OST efforts was 1994. 

All of the centers had award years of 1995 or later. 

In addition to forming divisions and technology centers, OST has also 

formed partnerships with governmental, public and private R&D 

organizations, agencies, and working groups. According to OST officials, 

an advantage of these partnerships is that OST can leverage the expertise 

and resources of already established R&D facilities without having to 

create their own. These partners have included 

OST’s Technical 

Partnerships for LongTerm Support 
Appendix V: OST’s Operations 

Page 41 GAO-04-198 Justice's Office of Science and Technology 

• corporations, such as Georgia Tech Research Corporation and L-3 

Communications, Analytics Corporation; 

• state and local agencies, such as the Houston Police Department and the 

Washington Metropolitan Area Transit Authority; 

• academic institutions, such as the University of Virginia and Syracuse 

University; 

• other federal government agencies, such as the Department of Defense’s 

Army Training and Doctrine Command, and the Department of 

Transportation’s Federal Aviation Administration; and 

• foreign government organizations, such as the Royal Canadian Mounted 

Police, the United Kingdom Police Scientific Development Branch, and the 

government of Israel. 

Each of OST’s technology centers is affiliated with one or more of OST’s 

technical partners. These technical partners are awarded funding in 

exchange for providing staff and facilities to the technology centers. Table 

9 lists OST’s partners and their affiliations, and funding they received to 

support the centers through June of fiscal year 2003. 
Appendix V: OST’s Operations 

Page 42 GAO-04-198 Justice's Office of Science and Technology 

Table 9: OST’s Technology Centers, Their Affiliated Partners, and the Amounts Awarded to Support the Centers 

Dollars in millions 

Technology centers Affiliated OST partner 

Amount awarded 

to support center 

Regional centers 

National, Rockville, Md. Aspen Systems Corporation, Rockville, Md. 20.4 

Northeast, Rome, N.Y. Air Force Research Laboratory, U.S. Air Force, Rome, N.Y. 11.7 

Southeast, North Charleston, S.C. South Carolina Research Authority, North Charleston, S.C. 21.3 

 Space and Naval Warfare Systems Center, U.S. Navy, 

Columbia, S.C. 0.6 

 Oak Ridge National Laboratory, U.S. Department of Energy, Oak 

Ridge, Tenn. 0.3 

 Savannah River Site, Department of Energy, Aiken, S.C. 1.3 

 

Northwest, Anchorage, Alaska Chenega Technology Services Corporation, and National 

Business Center, U.S. Department of Interior, Anchorage, Alaska 2.8 

Rocky Mountain, Denver, Colo. University of Denver - Colorado Seminary, Denver, Colo. 16.2 

West, El Segundo, Calif. Aerospace Corporation, El Segundo, Calif. 12.7 

Specialty centers 

Border Research Technology Center, San 

Diego, Calif 

Aerospace Corporation, El Segundo, Calif. 

1.4 

 Space and Naval Warfare Systems Center, U.S. Navy, San 

Diego, Calif. 1.7 

 Sandia National Laboratories, U.S. Department of Energy, 

Albuquerque, N. Mex. 5.1 

 U.S. Attorney’s Office, Southern District of California, 

Department of Justice, San Diego, Calif. 0.0a

Office of Law Enforcement Standards, 

Gaithersburg, Md. 

National Institute of Standards and Technology, U.S. Department 

of Commerce, Gaithersburg, Md. 53.6 

Office of Law Enforcement Technology 

Commercialization, Wheeling, W.Va. 

OLETC, Inc., Wheeling, W.Va. 

2.8 

 Wheeling Jesuit University, Wheeling, W.Va. 14.0 

 National Aeronautics and Space Administration 2.8 

Rural Law Enforcement Technology Center, 

Hazard, Ky. 

Eastern Kentucky University, Hazard, Ky. 

3.0 

Total fundingb $171.7 

Source: OST. 

Note: Figures are based on the current year values of each award. Award amounts are for the 

operations, maintenance and technical support of the centers. 

a

Actual amount is $25,000. 

b

Total might not add due to rounding. 
Appendix V: OST’s Operations 

Page 43 GAO-04-198 Justice's Office of Science and Technology 

To determine its program priorities, OST collaborates with its many 

customers and partners. Staff at both OST’s Washington, D.C., office and 

its technology centers are involved in helping OST to set program 

priorities. The staff report the results of their collaboration through formal 

meetings, periodic reports, and informal communication. Input is 

exchanged continually between OST’s customers and staff and within its 

multiple levels of organization. Using their input, the NIJ Director 

determines OST’s program priorities. (See figure 5 for the stakeholders, 

partners, and customers that contribute to the setting of OST’s priorities.) 

OST Collaborates with 

Many Customers and 

Partners to Determine 

Program Priorities 
Appendix V: OST’s Operations 

Page 44 GAO-04-198 Justice's Office of Science and Technology 

Figure 5: Stakeholders and Customers that Contribute to the Setting of OST’s 

Priorities 
Appendix V: OST’s Operations 

Page 45 GAO-04-198 Justice's Office of Science and Technology 

OST’s three divisions collaborate with other U.S. government agencies, the 

research and professional communities, and its technology centers to 

solicit input for setting priorities. Also, the divisions work with public 

safety practitioners at the state and local levels by, for example, meeting 

with grantees and assessing their needs. 

• The Investigative and Forensic Sciences Division collaborates with, 

and receives input from, researchers, academia, and the forensic 

laboratory community to help set program priorities. It also collaborates 

with, for example, the FBI and the interagency Technical Support Working 

Group. 

• The Research and Technology Development Division receives input 

through its collaboration with other federal agencies, such as the FBI, 

Drug Enforcement Administration, U.S. Secret Service, and White House 

Office of National Drug Control Policy. The division also participates in 

interagency working groups, such as for school safety and the Technical 

Support Working Group. Through these collaborations, OST can develop 

and share technologies used by both its customers and other agencies. For 

example, OST works with the Department of Defense to conduct less-thanlethal weapons R&D for law enforcement. 

• The Technology Assistance Division is primarily responsible for 

receiving input from OST’s technology centers. The centers solicit input 

from customers through their outreach efforts, such as technical 

assistance, e-mail exchanges, and telephone calls. The centers are also 

required to use OST’s web-based reporting system to record information 

on their customers’ requests for technical assistance. The centers are also 

required to submit monthly reports on their activities and finances. 

OST’s technology centers solicit input from the national and regional 

advisory councils that OST created to determine and advocate for the 

particular needs of its customers. Members of the national advisory 

council are selected by the technology centers and represent federal, state, 

and local public safety agencies, as well as international criminal justice 

organizations. Among its duties, this national advisory council identifies 

the present and future equipment and technology needs of OST’s 

customers and reviews the programs of the technology centers. In 

addition, the national advisory council recommends (1) ways to improve 

the technology centers’ programs’ relevance to the needs of the centers’ 

customers and (2) broad priorities for the technology center network and 

OST that are consistent with the needs of their customers. 

OST Collaborates with 

Government Agencies, 

Research and Professional 

Communities, and Centers 

Advisory Councils and 

Federal, State, and Local 

Public Safety Agencies 

Collaborate with OST’s 

Technology Centers 
Appendix V: OST’s Operations 

Page 46 GAO-04-198 Justice's Office of Science and Technology 

Each technology center has a regional advisory council. The regional 

advisory councils consist of a cross-section of law enforcement and other 

public safety officials who represent the interests of state and local 

officials. The regional advisory councils solicit input from the state and 

local agencies serviced in their regions, advise and support their 

respective center directors on their customers’ problems and needs, and 

advocate for resource support and improvements required by their 

customers. Through this method of sharing information, OST can better 

understand the needs of its customers. For example, OST’s regional 

councils can represent the unique needs of their customers that the 

national advisory council or the technology centers might not be aware of. 
Appendix VI: OST’s Goals in its Fiscal Year 

2004 Performance Plan and GAO’s 

Assessment 

Page 47 GAO-04-198 Justice's Office of Science and Technology 

Table 10: OST’s Performance Goals, Initiatives, and Measures for Fiscal Year 2004, and GAO’s Assessment 

 Type of measure 

OST’s initiatives Goals for initiatives 

Measures for assessing achievement of 

goals Output Intermediate Outcome 

1. Number of labs demonstrating improved 

access to external capabilities and 

increased lab capabilities. 

 X 

2. Number of samples (1) analyzed using the 

selected DNA markers that are required by 

the FBI’s national Combined DNA Index 

System (CODIS) database, and (2) made 

available for CODIS. 

 X 

1. Convicted 

Offender DNA 

Backlog 

Reduction 

Program 

Reduce DNA backlog 

and support a 

functioning, active 

system, which can solve 

old crimes and prevent 

new ones from occurring.

3. Number of states that have experienced an 

increase in the number of samples they 

have contributed to the national database. 

 X 

2. No Suspect 

DNA Backlog 

Reduction 

Program 

Reduce DNA backlog 

and support a 

functioning, active 

system, which can solve 

old crimes and prevent 

new ones from occurring.

4. Number of DNA samples from cases where 

there is no known suspect. 

 X 

3. Paul Coverdell 

National 

Forensic 

Sciences 

Improvement 

Grants 

Program 

Improve quality, 

timeliness, and credibility 

of forensic science 

services. 

5. Number of forensic labs with improved 

analytic and technological resources. 

 X 

6. Number of technology demonstrations and 

test indicators that describe the goods and 

services produced. 

X 

7. Number of prototype technologies 

developed. 

X 

8. Number of guides, standards, and 

assessments in progress. 

X 

9. Number of guides, standards, and 

assessments completed. 

X 

4. Critical Incident 

Response 

Technology 

Initiative 

Improve the ability of 

public safety responders, 

including law 

enforcement and 

corrections officers, to 

deal with critical 

incidents, save lives, and 

reduce property loss. 

10. Number of technologies introduced in law 

enforcement and corrections agencies. 

 X 

Appendix VI: OST’s Goals in its Fiscal Year 

2004 Performance Plan and GAO’s 

Assessment 
Appendix VI: OST’s Goals in its Fiscal Year 

2004 Performance Plan and GAO’s 

Assessment 

Page 48 GAO-04-198 Justice's Office of Science and Technology 

 Type of measure 

OST’s initiatives Goals for initiatives 

Measures for assessing achievement of 

goals Output Intermediate Outcome 

11. Number of projects researching new 

forensic DNA markers. 

X 

12. Number of development/validation studies 

for forensic DNA techniques. 

X 

13. Number of computer programs developed 

for forensic DNA analysis. 

X 

5. DNA Research 

& 

Development 

Develop faster and more 

powerful tools and 

techniques for the 

analysis of DNA 

evidence. These new 

tools and techniques will 

result in more crimes 

prevented and solved 

and more perpetrators 

brought to justice. 

14. Number of prototypes and tools for forensic 

DNA analysis. 

X 

15. Number of technology demonstrations and 

tests. 

X 

16. Number of prototype technologies 

developed. 

X 

17. Number of guides, standards, and 

assessments in progress. 

X 

18. Number of guides, standards, and 

assessments completed. 

X 

6. Law 

Enforcement 

Technology 

Research and 

Development 

Assist in applying 

technology to reduce the 

vulnerability of critical 

infrastructure; detect 

weapons and other 

contraband; improve 

technologies to locate 

and differentiate between 

individuals in structures; 

leverage information 

technology to enhance 

the responder 

community’s ability to 

anticipate and deal with 

critical incidents; identify 

and respond to terrorist 

attacks involving 

chemical, biological, and 

other unconventional 

weapons; and develop 

needed standards.a

19. Number of technologies introduced in law 

enforcement and corrections agencies. 

 X 

20. Number of technology demonstrations. X 

21. Number of conferences and forums. X 

7. School Safety 

Technology 

Assist school 

administrators and law 

enforcement in creating a 

safer and more 

productive learning 

environment. Safe, 

effective, appropriate, 

and affordable 

technologies can affect 

the perception and reality 

of safe schools. 

22. Number of school safety technology 

products. 

X 
Appendix VI: OST’s Goals in its Fiscal Year 

2004 Performance Plan and GAO’s 

Assessment 

Page 49 GAO-04-198 Justice's Office of Science and Technology 

 Type of measure 

OST’s initiatives Goals for initiatives 

Measures for assessing achievement of 

goals Output Intermediate Outcome 

23. Number of crime labs receiving specialized 

forensic services. 

X 

24. Number of capacity-building forensic R&D 

and validation projects funded. 

X 

25. Number of forensic technology training tools 

developed and distributed. 

X 

26. Number of labs providing continuing 

education or advanced training to crime 

analysts. 

X 

27. Number of crime labs with increased 

capacity for implementation of new forensic 

capabilities (including DNA analysis). 

 X 

28. Number of capacity-building forensic R&D 

and validation projects completed and 

impacting crime labs. 

 X 

29. Number of labs establishing new forensic 

capabilities. 

 X 

30. Number of labs expanding current forensic 

capabilities. 

 X 

31. Number of labs experiencing a reduction in 

time needed for evidence analysis. 

 X 

8. Crime Lab 

Improvement 

Program 

Provide immediate 

results in solving more 

crimes, bringing to justice 

more criminals, and 

improving administration 

of justice through the 

presentation of strong, 

reliable forensic evidence 

at trial. 

32. Number of labs experiencing a reduction in 

backlogged evidentiary sample analysis. 

 X 

33. Number of methods for examining 

evidentiary materials developed. 

X 

34. Number of standards for equipment and 

operating procedures developed. 

X 

9. Office for Law 

Enforcement 

Standards 

Help the public safety 

community make 

informed decisions about 

products being marketed 

for public safety 

personnel. 35. Law enforcement technology deliverables 

(standards, product performance 

evaluations, product guides). 

X 

36. Successful demonstration of prototype 

recognition system for smart gun. 

X 

37. Failure mode analysis for prototype 

recognition system for smart gun. 

X 

38. Incorporation and demonstration of 

recognition system into firearm (where 

applicable). 

X 

10. Smart Gun Develop a firearm that 

could save the lives of 

law enforcement officers 

and members of the 

public that they encounter 

while performing their 

duties. 

39. Identification of appropriate biometric 

solutions for recognition system (where 

applicable). 

X 
Appendix VI: OST’s Goals in its Fiscal Year 

2004 Performance Plan and GAO’s 

Assessment 

Page 50 GAO-04-198 Justice's Office of Science and Technology 

 Type of measure 

OST’s initiatives Goals for initiatives 

Measures for assessing achievement of 

goals Output Intermediate Outcome 

40. Number of technology information 

documents distributed. 

X 

41. Number of practitioners trained through the 

Crime Mapping Program. 

X 

11. OST’s network 

of technology 

centers 

(known as the 

National Law 

Enforcement 

and 

Corrections 

Technology 

Center 

system) 

Help state and local law 

enforcement, corrections, 

and public safety 

personnel do their jobs 

more safely and 

efficiently, thereby 

leading to greater 

administrative 

efficiencies, more crimes 

solved, and more lives 

saved. 

42. Savings to criminal justice agencies through 

the DOD’s Section 1033 Military Surplus 

Program. Section 1033 of the National 

Defense Authorization Act for Fiscal Year 

1997b

 authorizes DOD to transfer excess 

military property to federal and state 

agencies to support law enforcement 

activities including counterdrug and 

counterterrorism activities. 

 X 

Source: OST. 

a

Because the goal for this initiative was not outcome-oriented according to our criteria, we used the 

initiative’s mission statement as the goal. 

b

P.L. 104-201, 110 Stat. 2422 (1996). 
Appendix VII: Comments from the 

Department of Justice 

Page 51 GAO-04-198 Justice's Office of Science and Technology 

Appendix VII: Comments from the 

Department of Justice 
Appendix VII: Comments from the 

Department of Justice 

Page 52 GAO-04-198 Justice's Office of Science and Technology 
Appendix VII: Comments from the 

Department of Justice 

Page 53 GAO-04-198 Justice's Office of Science and Technology 
Appendix VIII: GAO Contacts and Staff 

Acknowledgments 

Page 54 GAO-04-198 Justice's Office of Science and Technology 

Laurie Ekstrand (202) 512-8777 

Weldon McPhail (202) 512-8644 

In addition to those named above, the following individuals contributed to 

this report: Samuel L. Hinojosa, Debra L. Picozzi, Katherine M. Davis, 

Richard Hung, Geoffrey R. Hamilton, Denise M. Fantone, Kristeen McLain, 

Elizabeth H. Curda, Rebecka Derr, Thomas M. Beall, and Leo M. Barbour. 

Appendix VIII: GAO Contacts and Staff 

Acknowledgments 

GAO Contacts 

Staff 

Acknowledgments 

(440165) 
The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal 

government for the American people. GAO examines the use of public funds; 

evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail 

this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail 

alerts” under the “Order GAO Products” heading. 

The first copy of each printed report is free. Additional copies are $2 each. A 

check or money order should be made out to the Superintendent of Documents. 

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a 

single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Public Affairs