Law Enforcement: Better Performance Measures Needed to Assess Results of Justice's Office of Science and Technology
Published: 2003-11-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The mission of the Office of Science & Technology (OST), within the Department of Justice's National Institute of Justice (NIJ), is to improve the safety and effectiveness of technology used by federal, state, and local law enforcement and other public safety agencies. Through NIJ, OST funds programs in forensic sciences, crime prevention, and standards and testing. To support these programs, Congress increased funding for OST from $13.2 million in 1995 to $204.2 million in 2003 (in constant 2002 dollars). GAO reviewed (1) the growth in OST's budgetary resources and the changes in OST's program responsibilities, (2) the types of products OST delivers and the methods used for delivering them; and (3) how well OST's efforts to measure the success of its programs in achieving intended results meet applicable requirements.
section_title: What GAO Found
paragraphs: OST's budgetary resources grew significantly in recent years, along with the range of its program responsibilities. From fiscal year 1995 through fiscal year 2003, OST received over $1 billion through Department of Justice appropriations and the reimbursement of funds from other federal agencies in exchange for OST's agreement to administer these agencies' projects. Of the over $1 billion that OST received, approximately $749 million, or 72 percent, was either directed to specific recipients or projects by public law, subject to guidance in congressional committee reports, or directed though reimbursable agreements. At the same time that spending expanded, OST's program responsibilities have changed--from primarily law enforcement and corrections to broader public safety technology. OST delivers three groups of products through various methods. The three groups include (1) information dissemination and technical assistance; (2) the application, evaluation, and demonstration of existing and new technologies for field users; and (3) technology research and development. According to OST, as of April 2003, it has delivered 945 products since its inception. Furthermore, OST identified an additional 500 products associated with ongoing awards. OST makes its products available through a variety of methods, such as posting information on its Web site and providing research prototypes to field users for testing and evaluation. OST has been unable to fully assess its performance in achieving its goals as required by applicable criteria because it does not use outcome measures to assess the extent to which it achieves the intended results of its programs. OST's current measures primarily track outputs, the goods and services produced, or in some cases OST uses intermediate measures, which is a step toward developing outcome measures. The Government Performance and Results Act of 1993 provides that federal agencies measure or assess the results of each program activity. While developing outcome measures for the types of activities undertaken by OST is difficult, we have previously reported on various strategies that can be used to develop outcome measures, or, at least intermediate measures, for similar types of activities.

section_title: Letter
paragraphs: To enhance public safety and bring criminals to justice, it is important for  law enforcement officials to benefit from the latest advances in science  and technology. The mission of the Office of Science and Technology  (OST), within the Department of Justice’s National Institute of Justice  (NIJ), is to improve the safety and effectiveness of technology used by  federal, state, and local law enforcement, corrections, and other public  safety agencies. OST awards funds to research and develop more effective  technology and improve access to technology in a wide range of areas. For  example, OST funds programs in the areas of crime prevention  technologies, investigative and forensic sciences, and electronic crime.  Examples of products resulting from OST’s programs include a guide on  school safety, an evaluation of police protective gear, a prototype for  ground-penetrating radar, and a report on gunshot residue detection and  interpretation. To support OST’s programs, Congress has significantly  increased its funding, from $13.2 million in fiscal year 1995 to $204.2  million in fiscal year 2003 (in constant 2002 dollars).
In response to your interest about whether OST’s programs are achieving  their intended results, we reviewed certain aspects of OST’s operations.  Specifically, this report assesses (1) the growth in OST’s budgetary  resources, from fiscal year 1995 to fiscal year 2003, and changes in OST’s  program responsibilities; (2) what types of products OST delivers and the  methods used to deliver these products to public safety agencies; and   (3) how well OST’s efforts to measure the success of its programs in  achieving intended results meet applicable requirements.
To address our objectives, we collected and analyzed relevant data and  reports and interviewed OST officials and NIJ officials, including NIJ  executive staff and the Assistant NIJ Director for OST, division chiefs, and  managers. We also collected data and interviewed officials at OST  technology centers in Rockville, Maryland; and El Segundo and San Diego,  California. Appendix I contains detailed information on the scope and  methodology we used for this assessment. We conducted this engagement  in accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The Office of Science and Technology (OST) was created in fiscal year  1995 following a long history of science and technology efforts within the  National Institute of Justice (NIJ). NIJ is a component of the Office of  Justice Programs (OJP), a Justice agency that, among other things,  provides assistance to state, tribal, and local governments. In establishing  OST’s objectives and allocating funds for OST’s programs, the NIJ Director  considers the priorities of many stakeholders, including the President,  Congress, Justice, and state and local law enforcement and public safety  agencies.
subsections: section_title: OST Established in Statute by the Homeland Security Act of 2002
paragraphs: In November 2002, Congress established OST and its mission and duties in  statute as part of the Homeland Security Act of 2002 (the Act). The Act  specified OST’s mission “to serve as the national focal point for work on  law enforcement technology; and to carry out programs that, through the  provision of equipment, training, and technical assistance, improve the  safety and effectiveness of law enforcement technology and improve  access to such technology by federal, state, and local law enforcement  agencies.” The Act defined the term “law enforcement technology” to  include “investigative and forensic technologies, corrections technologies,  and technologies that support the judicial process.” The Act also specified  OST’s duties to include the following, among others:    establishing and maintaining advisory groups to assess federal, state, and    establishing and maintaining performance standards, and testing,  evaluating, certifying, validating, and marketing products that conform to  those standards;   carrying out research, development, testing, evaluation, and cost-benefit  analysis of certain technologies; and   developing and disseminating technical assistance and training materials.
section_title: OST’s Operations
paragraphs: OST’s operations have multiple levels of internal organization and multiple  kinds of external partners. (For a more detailed description of OST’s  operations, see app. V.) OST’s multiple levels of organization include a  Washington, D.C., office and a network of 10 technology centers that  provide technical assistance to OST’s customers around the country. To  fulfill its mission, OST also collaborates with entities such as the  Departments of Defense and Energy and public and private laboratories to  take advantage of established technical expertise and resources.
NIJ has three main types of awards for funding OST’s programs: grants,  interagency agreements, and cooperative agreements.
Grants are generally awarded annually by NIJ to state and local agencies  or private organizations for a specific product and amount.
Interagency agreements are used by NIJ for creating partnerships with  federal agencies.
Cooperative agreements are a type of NIJ grant to nonfederal entities that  prescribes a higher level of monitoring and federal involvement.
NIJ also uses memorandums of understanding (MOU) to coordinate  programs and projects between agencies. The MOUs specify the roles,  responsibilities, and funding amounts to be provided by participating  agencies. Through NIJ, OST can provide supplemental funding to  interagency and cooperative agreements that may be used to contract for  special projects.
OST awards are administered by managers at its Washington, D.C., office  who have final oversight and management responsibility. These managers  may delegate some responsibility to another federal R&D agency receiving  the award. In March 2003, 21 managers were responsible for overseeing  336 active awards totaling $636 million.
Guidance has been established for measuring the performance of  government operations. To assist Justice to follow the Government  Performance and Results Act of 1993 (GPRA), OST establishes goals and  develops performance measures to track its progress. In addition, in May  2002, the White House Office of Management and Budget (OMB) and  Office of Science and Technology Policy issued a memorandum setting  forth R&D investment criteria that departments and agencies should  implement. The investment criteria require an explanation of why the  investment is important, how funds will be allocated to ensure quality, and  how well the investment is performing. According to the memorandum,  program managers must define appropriate outcome measures, and  milestones that can be used to track progress toward goals and assess  whether funding should be enhanced or redirected. The memorandum  encourages federal R&D agencies to make the processes they use to  satisfy GPRA consistent with these criteria.
section_title: OST’s Budgetary Resources Have Grown and Program Responsibilities Have Changed
paragraphs: OST’s budgetary resources have grown and the range of program  responsibilities has changed. Budgetary resources for OST increased  significantly, from $13.2 million in fiscal year 1995 to $204.2 million in  fiscal year 2003 (in constant 2002 dollars), totaling over $1 billion. This  increase can be attributed to the introduction of new allocations and large  increases for existing ones. The NIJ director decides how to allocate  certain appropriated funds to the various NIJ components, including OST.  About $749.7 million, or 72 percent, of OST’s total budgetary resources  was either directed to specific recipients or projects by public law, subject  to congressional committee report guidance designating specific  recipients or projects, or directed from the reimbursements from other  Justice and federal agencies in exchange for OST managing their projects.  Corresponding with the designation of spending for specific recipients and  projects, the range of OST’s programs changed, from primarily law  enforcement and corrections to include broader public safety technology  R&D, such as for improving school safety and combating terrorism.
subsections: section_title: Budgetary Resources for OST’s Programs
paragraphs: OST’s budgetary resources include both funding received via Justice  appropriations accounts as well as reimbursements from other Justice and  federal agencies. First, OST receives funding via three appropriations  accounts enacted in the appropriations law for the Justice Department.  From these appropriations accounts, OJP allocates amounts to NIJ. The  NIJ director suballocates part of the NIJ funds for OST programs. In  addition, OST receives reimbursements from other Justice and federal  agencies in exchange for OST’s management of specific projects of those  agencies, such as ballistic imaging evaluation for the FBI. Table 1 lists NIJ  allocations from the Justice appropriations accounts that go toward  funding OST programs.
OST’s budgetary resources almost quadrupled from fiscal year 1995 to  1996, increased 70 percent from fiscal year 1999 to 2000, and increased   63 percent from fiscal year 2001 to 2002. While resources decreased   24 percent from fiscal year 2002 to 2003, OST’s fiscal year 2003 level still  represents a 157 percent increase over the fiscal year 1999 level.
section_title: Certain Allocations Contributed to the Increase in Budgetary Resources since 1995
paragraphs: Our analysis of OST’s yearly budgetary resources from fiscal year 1995 to  fiscal year 2003 showed that the overall increase can be attributed to the  introduction of new NIJ allocations and large increases for existing ones.  The NIJ allocations that contributed to the overall increase in OST’s  budgetary resources are most notably the Crime Lab Improvement  Program, DNA Backlog Reduction, Safe Schools Technology R&D, and  Counterterrorism R&D allocations. Table 2 shows figures for all years in  constant 2002 dollars.
All dollar figures used in this narrative are in constant 2002 dollars, except  as noted otherwise.
Fiscal years 1995-1996: The $39.4 million (298 percent) increase from  $13.2 million to $52.6 million primarily came from two NIJ allocations  totaling $35.4 million.
Local Law Enforcement Block Grant (LLEBG) initiated with $22.2 million.    Reimbursement of funds increased by $13.2 million (471 percent) from  $2.8 million to $16.0 million.
Fiscal years 1999-2000: The $55.6 million (70 percent) increase from  $79.5 million to $135.1 million primarily came from three NIJ allocations  totaling $51.7 million.
DNA Backlog Reduction initiated with $15.6 million.    Safe Schools Technology R&D allocation initiated with $15.6 million.    Counterterrorism R&D increased by $20.5 million (193 percent) from $10.6  million to $31.1 million.
Fiscal years 2001-2002: The $103.4 million (63 percent) increase from  $164.6 million to $268.0 million primarily came from three NIJ allocations  totaling $95.3 million.
Reimbursement of funds increased by $55.6 million (209 percent) from  $26.6 million to $82.2 million.
DNA Backlog Reduction increased by $24.3 million (227 percent) from  $10.7 million to $35 million.
Crime Lab Improvement Program increased by $15.4 million (79 percent)  from $19.6 million to $35 million.
To be consistent with the report narrative and to show trends, figures in  table 2 are in constant 2002 dollars. A table with the figures in current  dollars can be found in appendix II.
OST had a $63.8 million (24 percent) decrease in total budgetary resources  from fiscal years 2002 to 2003, largely attributed to its not receiving fiscal  year 2003 Counterterrorism R&D resources, which totaled $45.3 million in  fiscal year 2002. According to OST, its counterterrorism resources were  transferred to the Department of Homeland Security’s Office of Domestic  Preparedness. There was also a $26.2 million decrease in the  reimbursement of funds from other agencies. However, OST’s fiscal year  2003 level still represents a 157 percent increase from fiscal year 1999.
section_title: Range of OST’s Program Responsibilities Has Changed
paragraphs: The range of OST’s program responsibilities has changed over the years  from primarily law enforcement and corrections to include broader public  safety technology R&D. This has happened as more and more of OST’s  budgetary resources were directed to be spent on specific recipients and  projects. Appropriated funds, for example, are sometimes designated for  specific recipients or projects in public law. In addition, guidance on the  spending of appropriated funds may be provided through congressional  committee reports. Of the more than $1 billion (in constant 2002 dollars)  that OST programs received from fiscal years 1995 to 2003, $532.6 million,  or 51 percent, was designated for specific recipients and projects in public  law or subject to guidance in committee reports designating specific  recipients or projects. Of the $532.6 million, $249.8 million was  designated in public law for specific recipients or projects while $282.8  million was specified in committee report guidance for specific recipients  or projects.
In addition to the $532.6 million designated in public law for specific  recipients or projects or subject to guidance in committee reports for  specific recipients or projects, another $217.1 million was reimbursements  from other Justice and federal agencies in exchange for OST’s  management of specific projects of those agencies. Thus, the total  spending either directed for specific recipients and projects through public  law, subject to committee report guidance designating specific recipients  or projects, or received as reimbursements, amounts to $749.7 million, or  72 percent, of OST’s total budgetary resources.
The range of OST’s program responsibilities has changed to include such  areas as school safety and counterterrorism. In fiscal year 1999, a Safe  Schools Initiative program was established pursuant to conference  committee report guidance with $10 million directing NIJ to develop  school safety technologies. In another example, OST’s counterterrorism  R&D program, initially funded by public law in fiscal year 1997, received  $147.3 million through fiscal year 2002, $96.6 million of which was  specified in conference report guidance for three recipients from fiscal  years 2000 to 2002—Oklahoma City National Memorial Institute for the  Prevention of Terrorism ($37.8 million), Dartmouth College’s Institute for  Security Technology Studies ($51.8 million), and the New York  University’s Center for Catastrophe Preparedness and Response   ($7 million).
OST’s program responsibilities have also changed to expand the focus on  investigative and forensic sciences. Our review of OST’s budgetary  resources for fiscal years 1995 through 2003 shows that budgetary  resources for investigative and forensic sciences equals at least   $342.1 million in constant fiscal year 2002 dollars, or about one-third, of  its $1 billion in budgetary resources, as shown in table 3. The proportion of  investigative and forensic sciences annual funding to total OST funding  rose from 6 percent ($800,000) in fiscal year 1995 to 52 percent   ($106.0 million) in fiscal year 2003.
section_title: OST Delivers Three Groups of Products Through Various Methods
paragraphs: OST delivers many products, which we categorized into three groups, and  uses various methods to deliver them. These three groups are   (1) information dissemination and technical assistance; (2) the  application, evaluation, and demonstration of existing and new  technologies for field users; and (3) technology R&D. According to OST, as  of April 2003, it had delivered 945 products since its inception.  Furthermore, OST identified an additional 500 products expected from  ongoing awards. Figure 2 shows our distribution of OST’s delivered  products by group. We recognize, as OST officials told us, that the groups  overlap and there is not a clean division between them. For example,  while reports are associated with information dissemination, they may  also result from the technology R&D group. OST has reviewed our  classification of products and agrees that it is generally accurate. Because  classification of some products is based on a judgment call, the  proportions of products in each group should be considered  approximations.
subsections: section_title: OST’s Range of Products
paragraphs: The following examples, while not exhaustive, indicate the wide range of  OST’s products.
Reports on topics such as analysis of DNA typing data, linguistic methods  for determining document authorship, a pepper spray projectile and  disperser, and gunshot residue detection and interpretation.
Prototypes of products including ground-penetrating radar, ballistics  matching using 3-dimensional images of bullets and cartridge cases, and  an optical recognition system to identify and track stolen vehicles.
Evaluations of technology including prison telemedicine networks, police  vehicles, and protective gear.
Guides on topics such as electronic crime scene investigation, use of  security technologies in schools, and antennas for radio communications.  For a more detailed description of OST’s products and further examples,  see appendix III.
section_title: Information Dissemination and Technical Assistance
paragraphs: Information dissemination and technical assistance represents about   63 percent of OST’s delivered products. OST provides information to its  customers in a variety of ways. For example, OST provides guidance to  R&D laboratories on the needs of public safety practitioners. To public  safety practitioners, OST recommends certain public safety practices,  tools, and technologies. Through its Office of Law Enforcement Standards,  OST develops performance standards to ensure that commercially  available public safety equipment, such as handheld and walk-through  metal detectors, meets minimum performance requirements. OST also  helps its customers enhance their technical capacities by providing them  with training and technical assistance through its Crime Lab Improvement  Program (which also provides supplies and equipment), DNA Backlog  Reduction Program, and network of technology centers. OST also uses the  R&D expertise and experience of already established laboratories and  other R&D organizations to provide additional guidance for managing  specialized technology projects. Further, OST helps its customers receive  surplus federal equipment by acting as their liaison to the equipment  transfer program of the Department of Defense. For example, equipment  transferred ranges from armored vehicles to boots and uniforms.
In addition, OST sponsors conferences, workshops, and forums that bring  together its customers, technologists, and policymakers. For example, it  sponsors the Mock Prison Riot, an annual event demonstrating emerging  technologies in riot training scenarios held at the former West Virginia  Penitentiary in Moundsville, West Virginia. This event brings together  corrections officers and vendors for technology showcases and training  exercises. Also, OST sponsors the Innovative Technologies for Community  Corrections Annual Conference, among others.
section_title: Application, Evaluation, and Demonstration of New and Existing Technologies
paragraphs: Another OST product group is the application, evaluation, and  demonstration of new and existing technologies, which represents about  20 percent of OST’s delivered products. Some of OST’s programs apply  existing technology solutions in new ways to assist public safety  operations. Examples of the application of new and existing technologies  include developing methods for the collection and analysis of chemical  trace evidence left from explosives and a handheld computer device  provided to bomb technicians in order to access bomb data at the scene of  incidents. In addition, OST tests commercially available products through  NIJ-certified laboratories to determine whether they are in accordance  with national performance standards. Examples of products evaluated  against standards include body armor, handcuffs, and semiautomatic  pistols. OST’s evaluations also include conducting field tests to compare  different commercially available products of the same type to allow users  to select the product that best suits their needs. OST also demonstrates  technology resulting from R&D directly to its customers through OST- sponsored events. For example, the Critical Incident Response  Technology Seminar, formerly known as the Operation America,  demonstrates live-fire simulation for bomb technicians. The annual Mock  Prison Riot demonstrates emerging technologies for use by corrections  officers and tactical team members.
section_title: Technology R&D
paragraphs: About 17 percent of OST’s delivered products were related to technology  R&D, which involves the development of prototype devices, among other  efforts. According to OST, R&D in its early stages includes development  of prototypes and demonstration that a principle can be proven. Applied  R&D, which also involves the development of prototypes, includes  technologies that are made available to public safety agencies, generally  through OST-assisted commercialization. Examples of products resulting  from OST’s applied R&D range from a bomb threat training simulator,  facial recognition technology for internet-based gang tracking, to a  personal alarm and location monitoring system for corrections officers.
According to OST, R&D in its early stages begins with testing technology  concepts, exploring solutions, and deciding whether continued  development is warranted. If OST decides to support product development  and if it has available funds, it awards funding to develop, demonstrate,  and evaluate an experimental prototype, which is then further developed  into an initial engineering prototype, and then demonstrated and  evaluated. If the prototype proves successful, OST demonstrates a “near  commercial” model to its customers for their evaluation.
While OST does not directly commercialize the results of its technology  R&D, it does provide prototypes to local users for field-testing and assists  in linking prototypes with potential commercial developers. OST officials  believe it would be a conflict of interest and therefore inappropriate for  them to promote one vendor or technology over another or try to dictate  what equipment their customers should purchase. OST’s role in  commercialization is to bring technologies and potential manufacturers  together so that the manufacturers can determine the feasibility of  commercializing the technologies.
section_title: OST’s Methods for Delivering its Products
paragraphs: OST delivers its products to its customers through a variety of methods.  (We recognize that products are sometimes delivery methods. For  example, a publication can be both a product resulting from research and  a method of information dissemination.) Besides publications, OST’s  methods for delivering information and technical assistance include mass  mailings; downloadable material from its Web site; panels, boards, and  working groups; training, support, and presentations; and programs to  enhance the capacity of public safety agencies.
OST also delivers its products related to application, evaluation, and  demonstration through various means. For example, private industry  provides new and existing technologies to OST; in turn, OST informs its  customers of the results of using these technologies in new ways. OST  publishes user guides and the test results of its evaluations of  commercially available equipment (both standards-based and comparison- based). Seeking to further educate its customers, OST demonstrates new  technology at technology fairs, providing “hands on” opportunities to use  it.
For its R&D products, OST may test “near commercial” prototypes in  particular settings. For example, OST may install in a police agency a  prototype technology that facilitates communications among public safety  agencies and across jurisdictions. If the technology is effective, the police  agency may incorporate the technology directly into its operations, before  the technology has become a commercial product.
section_title: OST’s Performance Measurement Efforts Do Not Fully Meet Requirements
paragraphs: OST’s efforts to measure its performance results, including the usefulness  and effectiveness of its products, do not fully meet applicable  requirements. To help Justice comply with the Government Performance  and Results Act of 1993 (GPRA), OST establishes goals and develops  performance measures to track its progress. GPRA, which mandates  performance measurements by federal agencies, requires, among other  things, that each agency measure or assess relevant outputs and outcomes  of each program activity. According to GPRA, the Office of Management  and Budget (OMB), and GAO, outcomes assess actual results as compared  with the intended results or consequences that occur from carrying out a  program or activity. Outputs count the goods and services produced by a  program or organization. Intermediate measures can be used to show  progress to achieving intended results. Subsequent OMB and committee  report guidance on GPRA, and previous GAO reports recognize that  output measures can provide important information in managing  programs. However, committee report guidance emphasizes using  outcome measures to aid policy makers because such measures are key to  assessing intended results.
subsections: section_title: OST Performance Measures Do Not Measure Results
paragraphs: The performance measures that OST has developed do not measure  results. According to the NIJ director, the Assistant Attorney General  (AAG) in April 2002 issued a memorandum requiring NIJ, including OST,  to develop outcome measures for fiscal year 2004. In August 2002, the NIJ  Director responded by stating that OST had indeed developed outcome  measures for its programs. In its fiscal year 2004 performance plan, OST  established goals for 11 of its initiatives and developed 42 measures for  assessing the achievement of those goals. However, based on our review  of OST’s performance plan, OMB guidance on GPRA, and GAO definitions  of outcome, output, and intermediate measures, we determined that of the  42 measures, none were outcome-oriented, 28 were output-oriented, and  14 were intermediate. See table 4 for GAO’s determination of the measures  and appendix VI for further details of our results.
According to Justice officials, R&D activities present measurement  challenges because outcomes are difficult or costly to measure. As the NIJ  Director pointed out, a May 2002, White House OMB and Office of Science  and Technology Policy memorandum concluded that agencies should not  have the same expectations for measuring the results of basic R&D as they  do for applied R&D. According to NIJ, relatively little of OST’s work is  basic R&D. As shown earlier, most of OST’s products are related to  information dissemination and technical assistance and the application,  evaluation, and demonstration of existing and new technologies for field  users.
We recognize that OST’s task in relation to measuring the results of even  non-basic research is complex in part because of the wide array of  activities it sponsors, and because of inherently difficult measurement  challenges involved in assessing the types of programs it undertakes. For  example, programs that are intended to deter crime face measurement  issues in assessing the extent to which something (crime) does not  happen. Nevertheless, improvement in measurement of program results is  important to help OST ensure it is doing all that is possible to achieve its  goals. It is worth noting that an outcome measure in relation to one OST  program was discussed by the NIJ Director in a May 2002 statement to  Congress. In this statement, the Director provided an example of an  outcome from the Convicted Offender DNA Backlog Reduction Program.  The Director stated that as a direct result of the program, approximately  400,000 convicted offender samples and almost 11,000 cases with no  suspect were analyzed. According to the NIJ Director, as of May 14, 2002,  more than 900 “hits” had been made on the FBI’s Combined DNA Index  System (CODIS) database as a direct result of the program, that is, 900  cases previously unsolved had been reopened. This information indicates  how the program is achieving its intended results in addressing unsolved  cases. Although this example seems to be a credible outcome measure, it  is not included in OST’s fiscal year 2004 performance plans.
section_title: Limitations in OST’s Efforts to Measure Effectiveness of Information Dissemination
paragraphs: OST efforts to measure information dissemination effectiveness have been  limited. One of the purposes of GPRA is to improve federal program  effectiveness and public accountability by promoting a new focus on  results, service quality, and customer satisfaction. Surveys to gauge  customer satisfaction represent one step toward finding out whether  customers have received information and whether they deem it of value.  However, these surveys have limitations in determining the extent to  which the information has been acted upon and resulted in intended  improvements. Thus, surveys such as these are more likely to be  intermediate measures (Did information get transferred?) than outcome  measures (Did information get transferred, acted upon, and achieve a  result?).
In 1998, NIJ initiated an effort to report the results of surveys to measure  the satisfaction of participants at all conferences, workshops, and seminar  series. OST reported on the “grantee level of satisfaction with NIJ  conferences” for fiscal years 1998-2000. However, in the fiscal years 2001- 2004 GPRA performance plans, OST discontinued tracking the surveys  because OJP and NIJ had ceased tracking these data as a performance  measure.
In fiscal year 2001, OST attempted to evaluate the effectiveness and value  of its TECHbeat newsletter. The survey sample of 5,500 was taken from a  distribution of major readership groups on TECHbeat’s mailing list of  20,674. According to OST, the response rate for the survey was too low to  produce statistically valid results: only 124 completed or substantially  completed responses were collected. The surveyors also experienced a  very low return on follow-up phone queries. According to the study, the  primary reason for the exceedingly low response rate was that so many  individuals on the mailing list had either changed jobs or were completely  unfamiliar with TECHbeat. Given these results, OST is trying to improve  the management and distribution of TECHbeat.
In fiscal year 2001, OST attempted to launch another effort to measure  program results, service quality, and customer satisfaction, but funding for  the effort was not provided. OST requested funding for an evaluation to  measure the success of its outreach efforts, including those by its  technology centers. The evaluation was to determine customer  satisfaction with its strategies for outreach and communication and with  its products. Specifically, OST planned to measure user satisfaction of the  content, format, and delivery mechanisms of its efforts, such as  technology information and assistance.
section_title: Most Studies of Other OST Initiatives Have Focused Primarily on Process
paragraphs: In fiscal years 1998 and 1999, OST funded eight outside studies of some of  its science and technology initiatives (see table 5). Our review of these  studies showed that seven of the eight studies focused on management  and organizational processes, and one was outcome-oriented.  Management and process evaluations can be useful tools for examining  how a program is operating and can offer insights into best practices. They  do not assess whether a program is achieving its intended results.
section_title: Efforts Are Under Way to Address Performance Measurement of Technology Centers
paragraphs: The Homeland Security Act of 2002 requires NIJ to transmit to Congress  by late November 2003 a report assessing the effectiveness of OST’s  existing system of technology centers and to identify the number of  centers necessary to meet the technology needs of federal, state, and local  law enforcement in the United States. According to NIJ, in response to the  Homeland Security Act requirement, it has initiated a study to assess the  impact and effectiveness of the technology center system and how it can  be enhanced to meet the evolving science and technology research and  technology needs of the state and local public safety community. NIJ also  stated that the report would address the functions that the technology  center system must provide to transfer NIJ’s research and development  results to practice in the criminal justice system. NIJ and OST have failed  to provide us with information detailing the methodology of the study, so  we cannot comment on the likelihood that this study will produce the  information sought by Congress. Additionally, according to OJP, the  technology centers are in the process of developing outcome measures to  demonstrate the impact of their activities.
According to NIJ, OJP has implemented additional performance measures  developed in May 2003 that will apply to NIJ, including OST. However, OJP  said it would defer implementing the measures related to the technology  centers until the results of the technology center study are known and NIJ  has a chance to take action, if warranted.
section_title: Measuring Results Is Difficult but Feasible
paragraphs: We acknowledge that measuring results using outcome measures is  difficult, and may be especially so in relation to some of the types of  activities undertaken by OST. Indeed, given the types and wide range of  program goals for OST efforts—solving old crimes, saving lives, and  reducing property loss—it may be the case that for some programs  intermediate measures represent the best feasible measure of results. We  note that approximately 63 percent of OST’s products fall into the category  of information dissemination and technical assistance, aimed at informing  customers and ultimately encouraging adoption of research results that  lead to increased efficiency and effectiveness. There are strategies  available that have been used by other federal agencies to take steps  toward assessing the effectiveness of information dissemination and  technical assistance efforts. For example, a recent GAO report outlines  various strategies to assess media campaigns and informational seminars,  including immediate post workshop surveys and follow-up surveys and the  use of logic models to define measures of a program’s progress toward  intended results and long-term goals.
section_title: Conclusions
paragraphs: Given the wide range of its products, OST has the potential to significantly  improve the technological capabilities of federal, state, and local public  safety agencies. However, the lack of information about the results of  program efforts, or the assessment of progress toward goals, means that  little is known about their effectiveness. While developing outcome  measurements in many research-related programs is extremely difficult,  there are various performance measurement strategies that other federal  programs have used for assessing information dissemination, technical  assistance and other R&D activities that might be applied to OST’s  programs. It is important to develop outcome measurements where  feasible, or intermediate measurements where appropriate, to assist  Congress, OST and NIJ management, and OST’s customers to better assess  whether investment in OST’s programs is paying off with improved law  enforcement and public safety technology.
section_title: Recommendation
paragraphs: To help ensure that OST does all that is possible to measure its progress in  achieving goals through outcome-oriented measures, we recommend that  the Attorney General instruct the Director of NIJ to reassess the measures  OST uses to evaluate its progress toward achieving its goals and to better  focus on outcome measures to assess results where possible. In those  cases where measuring outcome is, after careful consideration, deemed  infeasible, we recommend developing appropriate intermediate measures  that will help to discern program effectiveness.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a copy of a draft of this report to the Attorney General of the  United States for review and comment. In an October 30, 2003, letter, the  Assistant Attorney General (AAG) for OJP commented on the draft. Her  comments are summarized below and presented in their entirety in  appendix VII. OJP also provided technical comments, which have been  incorporated into this report where appropriate.
In the AAG’s written response, the Justice Department concurred with our  recommendation that NIJ reassess the measures OST uses to assess  program outcomes. In response to our recommendation, the AAG reported  that she has directed the NIJ Director, to reassess NIJ’s performance  measures for OST and to refine them, where possible, in order to focus  them more toward measuring outcomes.
While the AAG agreed with our recommendation, she also made several  other comments. First, she commented that developing numerical  outcome measures like those urged by GAO is a particular challenge for  R&D activities. As stated in our report, we recognize that measuring  results using outcome measures is difficult and may be especially so in  relation to some of the types of activities undertaken by OST. Our  reference to a numerical measure is meant only as an example of how one  of OST’s program activities can be linked to intended results. We believe  that further consideration of measures, both quantitative and qualitative,  could improve the assessment of results for R&D as well as other OST  programs. Our report also notes that relatively little of OST’s work is R&D.  The majority of OST’s products are in the category of information  dissemination and technical assistance.
Second, the AAG noted that GAO did not reach any conclusions in its  discussion of OST’s growth in budgetary resources, changes in program  responsibilities, management of programs, and delivery of its products.  The AAG noted that Justice believed that OST’s record is outstanding.  Neither OST nor we can determine whether OST’s efforts in these areas  are successful or otherwise, given that OST has not developed measures to  assess their outcomes. Therefore, it is not possible to draw conclusions.
Third, the AAG indicated that GAO did not discuss in detail that over one- half of OST’s funds were designated by Congress for specific recipients  and projects. She noted that GAO missed an opportunity to inform the  requester of the impact of Congress’ recent decisions regarding OST. We  reported that 51 percent of OST’s budgetary resources were designated for  specific recipients and projects in public law or subject to guidance in  committee reports.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 10 days from its  issue date. At that time, we will send copies of the report to the Attorney  General, appropriate congressional committees and other interested  parties. We will also make copies available to others upon request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov. Major contributors to this report are listed in  appendix VIII.
If you or your staff have any questions concerning this report, contact me  on (202) 512-8777.
section_title: Appendix I: Scope and Methodology
paragraphs: To answer our objectives, we interviewed National Institute of Justice  (NIJ) and Office of Science and Technology (OST) officials and collected  documents at OST’s office in Washington, D.C., and at three of OST’s  technology centers—the National center in Rockville, Maryland; West  center in El Segundo, California; and Border Research and Technology  Center in San Diego, California. We selected the Rockville center because  of its proximity to Washington, D.C., and the other two centers because of  their locations and particular areas of technology and technical  concentrations. We also interviewed a small group of OST’s customers— federal, state, and local law enforcement, and corrections and public  safety officials—who were selected by officials at the El Segundo and San  Diego centers. In addition, we analyzed information that is available on the  National Institute of Justice’s public Web site.
To determine OST’s budgetary resources and amounts from fiscal year  1995 to fiscal year 2003 and the changes in OST program responsibilities,  we reviewed NIJ and OST budget documents, interviewed officials in  OST’s Technology Management and the OJP’s Office of Budget and  Management Services, and reviewed pertinent appropriations laws and  committee reports covering that period. To determine the amount of OST  budgetary resources that were directed to specific recipients and projects,  we compared OST’s budget documents that listed individual recipients and  projects with the public laws and reports. We defined directed spending as  spending for specific recipients and projects designated in appropriations  laws or subject to congressional committee report guidance designating  specific recipients or projects. We did not determine the amount of  reimbursable projects designated in public laws or specified in committee  reports because those projects were not originally allocated to OST.  Instead, we considered all the reimbursable projects to be specific  projects for which OST was directed pursuant to its agreements with other  agencies on spending its reimbursable funds.
To determine the changes in OST’s program responsibilities, we analyzed  the year-to-year changes in its budget and program scope. To determine  the amount of OST’s budgetary resources used for investigative and  forensic sciences for fiscal years 1995-2003, we compared OST’s portfolio  description and NIJ’s definition of forensic sciences with the individual  budget program and project items listed in OST’s budget documents for  each fiscal year. However, while we recognize that OST’s technology  centers and their technical partners include investigative and forensic  sciences in their provision of technical assistance, we did not attempt to  determine the amount of center funds associated with investigative and  forensic sciences because the budget documents we received from OST  did not break out such amounts within the funding awarded to the centers.  Therefore, our determination that $342.1 million of OST’s total funding  supported investigative and forensic sciences did not include such  amounts.
To determine the amounts of funding awarded to the technology centers,  we analyzed databases on all of the products OST has produced through  April 2003 and the associated grants, interagency agreements, and  cooperative agreements and their amounts.
To determine the composition of OST’s products and how OST delivers  the products to its customers, we analyzed OST documents and a database  of all the products associated with its past and ongoing awards, from  inception through April 2003, that were delivered or anticipated to be  delivered. While the database included the award amounts associated with  the products, it was not possible to reliably associate the award amounts  for each product or type of product because multiple types of products  could result from individual awards. We also conducted interviews with  the parties mentioned above.
For the budget and product data that OST provided us, we assessed the  reliability of these data by examining relevant fields for missing data,  conducting range checks to identify any potentially erroneous outliers and  inspecting a subset of selected data elements that were common to two or  more data sets. In addition, we independently verified selected budget  data back to appropriations legislation and Committee reports. In  conducting our analyses, we identified  some potential data errors or  reliability problems. When this occurred, we contacted agency officials to  address and resolve these matters. However, we did not verify the budget  or product data back to source materials. Overall, we determined that  budget or product data provided to us is adequate for the descriptive  purposes it is used in this report.
We examined OST’s efforts to measure performance by interviewing  officials on this matter at OJP, NIJ, and OST in the Washington, D.C.,  office along with officials and staff at the technology centers, and current  and previous Advisory Council officials. We also reviewed related agency  documents, such as the OJP mission statement and performance plans;  NIJ strategic planning documents and website pages, annual performance  plans and performance reports, and GPRA documents; policies and  procedures, Department of Justice memoranda, OST internal planning and  reporting documents, program descriptions and documentation, and other  related documents.
As part of our examination, we reviewed OST’s fiscal year 1997 to 2004  goals and measures as presented in OST’s GPRA performance plans. We  focused our review on OST’s fiscal year 2004 performance plan and  measures. As part of our review of these goals and measures, we made a  determination as to whether the performance measure was output,  outcome, or intermediate-oriented. To make this determination about the  types of performance measures contained in OST’s performance plans, we  compared the measures used in the plans with the requirements of GPRA,  accompanying committee report, OMB’s guidance on performance  measurement challenges (Circular A-11), Justice’s guidance to its  components for preparing performance measures, and previous GAO work  on GPRA.
Also included in our examination of OST performance measurement  efforts were studies prepared by external parties under grants from OST  that reviewed selected OST initiatives such as portfolio areas, projects,  and programs. In response to our request for all of OST’s efforts to assess  its programs, OST provided eight outside studies funded from fiscal years  1998 to 1999. For example, the Pymatuning Group, Inc., conducted an  “Assessment of the National Law Enforcement and Corrections  Technology Center (NLECTC) Program,” which described the operations  of the OST’s regional technology centers network. We reviewed all eight of  the outside studies for performance information on the OST initiatives  being examined in the report. We examined the studies to determine  whether they provided information that would be considered consistent  with an outcome-oriented evaluation as defined by our criteria.
The scope of this review was limited to OST, and therefore we cannot  compare OST’s efforts to measure the performance of its programs or the  amount of funding directed to specific recipients and projects with the  efforts and funding of any other federal R&D agencies. We performed our  audit work from September 2002 to September 2003 in Washington, D.C.,  and other cited locations in accordance with generally accepted  government auditing standards.
section_title: Appendix II: Bugetary Resources for OST’s Programs in Current Year Dollars
subsections: section_title: Reimbursements from other Justice and federal agencies
section_title: Appendix III: OST’s 10 Categories of Products
paragraphs: While we divided OST’s products into three groups for our reporting  purposes, OST divides them into 10 categories. (See table 7 for GAO’s   3 groupings of OST’s 10 categories.) In regrouping OST’s 10 categories, we  recognized, as OST officials told us, that the 10 categories overlap and  there is not a clean division between them. We also recognized that many  of OST’s products could also be considered a delivery method. For  example, publications, such as the TECHbeat newsletter, are OST  products that can also represent a method of delivery for OST technology  information. OST has reviewed our classification of products and agrees  that it is generally accurate.
section_title: Appendix IV: OST’s Portfolio Areas
paragraphs: OST has organized its individual projects to develop, improve, and  implement technology for public safety agencies into nine portfolio areas.  As of April 2003, these portfolio areas included    critical incident technology, for first responders and investigators  protecting the public in the event of critical incidents such as natural  disasters, industrial accidents, or terrorist acts;   communications interoperability and information sharing,  enhancing communication among public safety agencies through wired  links, wireless radios, and information networks, even when disparate  systems are involved;   electronic crime, supporting computer forensic laboratories, publishing  guides for handling electronic evidence, and developing computer forensic  tools; investigative and forensic sciences, funding at the state and local levels  for DNA-typing of convicted offenders and use of DNA-typing in the  investigation of unsolved cases, and developing tools for forensic  casework;   crime prevention technologies, including contraband detectors, sensors  and surveillance systems, and biometric technologies;   protective systems technologies, including body armor; “smart”  handguns, which fire only upon recognition of, for example, a certain  handprint or password; puncture resistant gloves; better handcuffs; better  concealed weapon detection; and personnel tracking and location  technologies;  less-than-lethal technologies, developing alternatives to lethal force,  including technologies involving electrical or chemical effects, light  barriers, vehicle stopping, and blunt trauma, and evaluating and modeling  the effects of these technologies; learning technologies, developing technology tools for agencies to use in  training their personnel, including use of the internet, CD-ROMs, and  video-based and interactive simulations; and    standards and testing, ensuring that the equipment public safety  agencies buy is safe, dependable, and effective.
section_title: Appendix V: OST’s Operations
paragraphs: As with other federal agencies, OST’s operations involve multiple levels of  internal organization and multiple kinds of external partners. OST’s  multiple levels of organization include a Washington, D.C., office that  manages its technology programs and a network of technology centers  around the country that provide technical assistance to OST’s regional  customers. OST also collaborates with other R&D entities, such as those in  the Departments of Defense and Energy and public and private  laboratories, by forming technical partnerships in order to leverage  already established technical expertise and resources to support their  program efforts. Another aspect of OST’s complex operations is the need  to determine OST’s own priorities and the priorities of its customers,  which involves Washington and regional center staff collaborating  formally and informally with a myriad of federal, state, and local officials,  as well as with one another.
subsections: section_title: OST Has Multiple Levels of Organization
paragraphs: OST’s multiple levels of organization include a Washington, D.C., office  and technology centers, as well as technical partnerships with  government, public and private R&D and public safety organizations. As of  September 2003, OST’s Washington office consisted of 25 full-time- equivalent Justice staff divided into three divisions and under the Assistant  NIJ Director for OST. Responsibility for managing these programs is  divided among the three divisions. (See figure 3 for OST’s organizational  structure.)
Research and Technology Development Division manages electronic  crime (including cybercrime), critical incidents and counterterrorism,  communications interoperability and information sharing, crime  prevention, learning technology tools, less-than-lethal technologies,  standards development, school safety, and corrections technologies.
Investigative and Forensic Sciences Division manages DNA-related  R&D and other investigative and forensic sciences, such as fingerprint  analysis, and includes the Crime Laboratory Improvement Program  projects, DNA Backlog Reduction projects, and DNA research and  development projects.
Technology Assistance Division, through the technology center  network, provides training and technical advice to, and identifies  technologies for, OST’s customers, and oversees OST’s network of 10  technology centers (see figure 4). The technology centers are another  source of technical advice for OST’s customers.
section_title: OST’s Technology Centers
paragraphs: OST’s network of 10 technology centers provides technical assistance,  among other things, to OST’s customers. From fiscal year 1995 to fiscal  year 2003 (as of July 2003), funding support for the centers totaled   $171.7 million. (See table 8 for funding by center.) The technology centers  comprise six regional centers and four specialty centers. While the  regional centers assist OST’s customers by region—Northwest, West,  Rocky Mountain, Northeast, Southeast, and National—they are expected  to coordinate and collaborate among one another regardless of where the  resources and capabilities are located. Each of these 6 centers works with  a regional advisory council comprising state and local law enforcement,  corrections, and public safety representatives.
As described below, the four specialty centers provide specialized  expertise and services.
The Office of Law Enforcement Standards tests commercially available  equipment and develops minimum performance standards for such  equipment.
The Office of Law Enforcement Technology Commercialization, Inc.,  assists inventors and developers, among others, in commercializing  technologies.
The Border Research and Technology Center aids in the development of  technologies for agencies concerned with law enforcement at the northern  and southern borders.
The Rural Law Enforcement Technology Center aids rural and small- community law enforcement and corrections agencies.
section_title: OST’s Technical Partnerships for Long- Term Support
paragraphs: In addition to forming divisions and technology centers, OST has also  formed partnerships with governmental, public and private R&D  organizations, agencies, and working groups. According to OST officials,  an advantage of these partnerships is that OST can leverage the expertise  and resources of already established R&D facilities without having to  create their own. These partners have included    corporations, such as Georgia Tech Research Corporation and L-3  Communications, Analytics Corporation;   state and local agencies, such as the Houston Police Department and the  Washington Metropolitan Area Transit Authority;   academic institutions, such as the University of Virginia and Syracuse    other federal government agencies, such as the Department of Defense’s  Army Training and Doctrine Command, and the Department of  Transportation’s Federal Aviation Administration; and foreign government organizations, such as the Royal Canadian Mounted  Police, the United Kingdom Police Scientific Development Branch, and the  government of Israel.
Each of OST’s technology centers is affiliated with one or more of OST’s  technical partners. These technical partners are awarded funding in  exchange for providing staff and facilities to the technology centers. Table  9 lists OST’s partners and their affiliations, and funding they received to  support the centers through June of fiscal year 2003.
section_title: OST Collaborates with Many Customers and Partners to Determine Program Priorities
paragraphs: To determine its program priorities, OST collaborates with its many  customers and partners. Staff at both OST’s Washington, D.C., office and  its technology centers are involved in helping OST to set program  priorities. The staff report the results of their collaboration through formal  meetings, periodic reports, and informal communication. Input is  exchanged continually between OST’s customers and staff and within its  multiple levels of organization. Using their input, the NIJ Director  determines OST’s program priorities. (See figure 5 for the stakeholders,  partners, and customers that contribute to the setting of OST’s priorities.)
section_title: OST Collaborates with Government Agencies, Research and Professional Communities, and Centers
paragraphs: OST’s three divisions collaborate with other U.S. government agencies, the  research and professional communities, and its technology centers to  solicit input for setting priorities. Also, the divisions work with public  safety practitioners at the state and local levels by, for example, meeting  with grantees and assessing their needs.
The Investigative and Forensic Sciences Division collaborates with,  and receives input from, researchers, academia, and the forensic  laboratory community to help set program priorities. It also collaborates  with, for example, the FBI and the interagency Technical Support Working  Group.
The Research and Technology Development Division receives input  through its collaboration with other federal agencies, such as the FBI,  Drug Enforcement Administration, U.S. Secret Service, and White House  Office of National Drug Control Policy. The division also participates in  interagency working groups, such as for school safety and the Technical  Support Working Group. Through these collaborations, OST can develop  and share technologies used by both its customers and other agencies. For  example, OST works with the Department of Defense to conduct less-than- lethal weapons R&D for law enforcement.
The Technology Assistance Division is primarily responsible for  receiving input from OST’s technology centers. The centers solicit input  from customers through their outreach efforts, such as technical  assistance, e-mail exchanges, and telephone calls. The centers are also  required to use OST’s web-based reporting system to record information  on their customers’ requests for technical assistance. The centers are also  required to submit monthly reports on their activities and finances.
section_title: Advisory Councils and Federal, State, and Local Public Safety Agencies Collaborate with OST’s Technology Centers
paragraphs: OST’s technology centers solicit input from the national and regional  advisory councils that OST created to determine and advocate for the  particular needs of its customers. Members of the national advisory  council are selected by the technology centers and represent federal, state,  and local public safety agencies, as well as international criminal justice  organizations. Among its duties, this national advisory council identifies  the present and future equipment and technology needs of OST’s  customers and reviews the programs of the technology centers. In  addition, the national advisory council recommends (1) ways to improve  the technology centers’ programs’ relevance to the needs of the centers’  customers and (2) broad priorities for the technology center network and  OST that are consistent with the needs of their customers.
Each technology center has a regional advisory council. The regional  advisory councils consist of a cross-section of law enforcement and other  public safety officials who represent the interests of state and local  officials. The regional advisory councils solicit input from the state and  local agencies serviced in their regions, advise and support their  respective center directors on their customers’ problems and needs, and  advocate for resource support and improvements required by their  customers. Through this method of sharing information, OST can better  understand the needs of its customers. For example, OST’s regional  councils can represent the unique needs of their customers that the  national advisory council or the technology centers might not be aware of.
section_title: Appendix VI: OST’s Goals in its Fiscal Year 2004 Performance Plan and GAO’s Assessment
paragraphs: Reduce DNA backlog  and support a  functioning, active  system, which can solve  old crimes and prevent  new ones from occurring.
1.  Number of labs demonstrating improved  access to external capabilities and  increased lab capabilities.  2.  Number of samples (1) analyzed using the  selected DNA markers that are required by  the FBI’s national Combined DNA Index  System (CODIS) database, and (2) made  available for CODIS.  3.  Number of states that have experienced an  increase in the number of samples they  have contributed to the national database.
Reduce DNA backlog  and support a  functioning, active  system, which can solve  old crimes and prevent  new ones from occurring.
4.  Number of DNA samples from cases where  there is no known suspect.
Improve quality,  timeliness, and credibility  of forensic science  services.
5.  Number of forensic labs with improved  analytic and technological resources.
Improve the ability of  public safety responders,  including law  enforcement and  corrections officers, to  deal with critical  incidents, save lives, and  reduce property loss.
6.  Number of technology demonstrations and  test indicators that describe the goods and  services produced.  7.  Number of prototype technologies  developed.  8.  Number of guides, standards, and  assessments in progress.  9.  Number of guides, standards, and  assessments completed.  10. Number of technologies introduced in law  enforcement and corrections agencies.
Develop faster and more  powerful tools and  techniques for the  analysis of DNA  evidence. These new  tools and techniques will  result in more crimes  prevented and solved  and more perpetrators  brought to justice.
11. Number of projects researching new  forensic DNA markers.  12. Number of development/validation studies  for forensic DNA techniques.  13. Number of computer programs developed  for forensic DNA analysis.  14. Number of prototypes and tools for forensic  DNA analysis.
Assist in applying  technology to reduce the  vulnerability of critical  infrastructure; detect  weapons and other  contraband; improve  technologies to locate  and differentiate between  individuals in structures;  leverage information  technology to enhance  the responder  community’s ability to  anticipate and deal with  critical incidents; identify  and respond to terrorist  attacks involving  chemical, biological, and  other unconventional  weapons; and develop  needed standards.
15. Number of technology demonstrations and  tests.  16. Number of prototype technologies  developed.  17. Number of guides, standards, and  assessments in progress.  18. Number of guides, standards, and  assessments completed.  19. Number of technologies introduced in law  enforcement and corrections agencies.  20. Number of technology demonstrations.
Assist school  administrators and law  enforcement in creating a  safer and more  productive learning  environment. Safe,  effective, appropriate,  and affordable  technologies can affect  the perception and reality  of safe schools.
21. Number of conferences and forums.  22. Number of school safety technology  products.
Provide immediate  results in solving more  crimes, bringing to justice  more criminals, and  improving administration  of justice through the  presentation of strong,  reliable forensic evidence  at trial. forensic services.  24. Number of capacity-building forensic R&D  and validation projects funded.  25. Number of forensic technology training tools  developed and distributed.  26. Number of labs providing continuing  education or advanced training to crime  analysts.  27. Number of crime labs with increased  capacity for implementation of new forensic  capabilities (including DNA analysis).  28. Number of capacity-building forensic R&D  and validation projects completed and  impacting crime labs.  29. Number of labs establishing new forensic  capabilities.  30. Number of labs expanding current forensic  capabilities.  31. Number of labs experiencing a reduction in  time needed for evidence analysis.  32. Number of labs experiencing a reduction in  backlogged evidentiary sample analysis.
Help the public safety  community make  informed decisions about  products being marketed  for public safety  personnel.
33. Number of methods for examining  evidentiary materials developed.  34. Number of standards for equipment and  operating procedures developed.  35. Law enforcement technology deliverables  (standards, product performance  evaluations, product guides).
Develop a firearm that  could save the lives of  law enforcement officers  and members of the  public that they encounter  while performing their  duties.
36. Successful demonstration of prototype  recognition system for smart gun.  37. Failure mode analysis for prototype  recognition system for smart gun.  38. Incorporation and demonstration of  recognition system into firearm (where  applicable).  39. Identification of appropriate biometric  solutions for recognition system (where  applicable).
Help state and local law  enforcement, corrections,  and public safety  personnel do their jobs  more safely and  efficiently, thereby  leading to greater  administrative  efficiencies, more crimes  solved, and more lives  saved.
40. Number of technology information  documents distributed.  41. Number of practitioners trained through the  Crime Mapping Program.  42. Savings to criminal justice agencies through  the DOD’s Section 1033 Military Surplus  Program. Section 1033 of the National  Defense Authorization Act for Fiscal Year  1997 authorizes DOD to transfer excess  military property to federal and state  agencies to support law enforcement  activities including counterdrug and  counterterrorism activities.
section_title: Appendix VII: Comments from the Department of Justice
section_title: Appendix VIII: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, the following individuals contributed to  this report: Samuel L. Hinojosa, Debra L. Picozzi, Katherine M. Davis,  Richard Hung, Geoffrey R. Hamilton, Denise M. Fantone, Kristeen McLain,  Elizabeth H. Curda, Rebecka Derr, Thomas M. Beall, and Leo M. Barbour.