Law Enforcement: Better Performance Measures Needed to Assess Results of Justice's Office of Science and Technology
Released: 2003-12-09T00:00:00

Letter

Background

The Office of Science and Technology (OST) was created in fiscal year 1995 following a long history of science and technology efforts within the National Institute of Justice (NIJ). NIJ is a component of the Office of Justice Programs (OJP), a Justice agency that, among other things, provides assistance to state, tribal, and local governments. In establishing OST’s objectives and allocating funds for OST’s programs, the NIJ Director considers the priorities of many stakeholders, including the President, Congress, Justice, and state and local law enforcement and public safety agencies.

OST Established in Statute by the Homeland Security Act of 2002

In November 2002, Congress established OST and its mission and duties in statute as part of the Homeland Security Act of 2002 (the Act). The Act specified OST’s mission “to serve as the national focal point for work on law enforcement technology; and to carry out programs that, through the provision of equipment, training, and technical assistance, improve the safety and effectiveness of law enforcement technology and improve access to such technology by federal, state, and local law enforcement agencies.” The Act defined the term “law enforcement technology” to include “investigative and forensic technologies, corrections technologies, and technologies that support the judicial process.” The Act also specified OST’s duties to include the following, among others: establishing and maintaining advisory groups to assess federal, state, and establishing and maintaining performance standards, and testing, evaluating, certifying, validating, and marketing products that conform to those standards; carrying out research, development, testing, evaluation, and cost-benefit analysis of certain technologies; and developing and disseminating technical assistance and training materials.

OST’s Operations

OST’s operations have multiple levels of internal organization and multiple kinds of external partners. (For a more detailed description of OST’s operations, see app. V.) OST’s multiple levels of organization include a Washington, D.C., office and a network of 10 technology centers that provide technical assistance to OST’s customers around the country. To fulfill its mission, OST also collaborates with entities such as the Departments of Defense and Energy and public and private laboratories to take advantage of established technical expertise and resources.

NIJ has three main types of awards for funding OST’s programs: grants, interagency agreements, and cooperative agreements.

Grants are generally awarded annually by NIJ to state and local agencies or private organizations for a specific product and amount.

Interagency agreements are used by NIJ for creating partnerships with federal agencies.

Cooperative agreements are a type of NIJ grant to nonfederal entities that prescribes a higher level of monitoring and federal involvement.

NIJ also uses memorandums of understanding (MOU) to coordinate programs and projects between agencies. The MOUs specify the roles, responsibilities, and funding amounts to be provided by participating agencies. Through NIJ, OST can provide supplemental funding to interagency and cooperative agreements that may be used to contract for special projects.

OST awards are administered by managers at its Washington, D.C., office who have final oversight and management responsibility. These managers may delegate some responsibility to another federal R&D agency receiving the award. In March 2003, 21 managers were responsible for overseeing 336 active awards totaling $636 million.

Guidance has been established for measuring the performance of government operations. To assist Justice to follow the Government Performance and Results Act of 1993 (GPRA), OST establishes goals and develops performance measures to track its progress. In addition, in May 2002, the White House Office of Management and Budget (OMB) and Office of Science and Technology Policy issued a memorandum setting forth R&D investment criteria that departments and agencies should implement. The investment criteria require an explanation of why the investment is important, how funds will be allocated to ensure quality, and how well the investment is performing. According to the memorandum, program managers must define appropriate outcome measures, and milestones that can be used to track progress toward goals and assess whether funding should be enhanced or redirected. The memorandum encourages federal R&D agencies to make the processes they use to satisfy GPRA consistent with these criteria.

OST’s Budgetary Resources Have Grown and Program Responsibilities Have Changed

OST’s budgetary resources have grown and the range of program responsibilities has changed. Budgetary resources for OST increased significantly, from $13.2 million in fiscal year 1995 to $204.2 million in fiscal year 2003 (in constant 2002 dollars), totaling over $1 billion. This increase can be attributed to the introduction of new allocations and large increases for existing ones. The NIJ director decides how to allocate certain appropriated funds to the various NIJ components, including OST. About $749.7 million, or 72 percent, of OST’s total budgetary resources was either directed to specific recipients or projects by public law, subject to congressional committee report guidance designating specific recipients or projects, or directed from the reimbursements from other Justice and federal agencies in exchange for OST managing their projects. Corresponding with the designation of spending for specific recipients and projects, the range of OST’s programs changed, from primarily law enforcement and corrections to include broader public safety technology R&D, such as for improving school safety and combating terrorism.

Budgetary Resources for OST’s Programs

OST’s budgetary resources include both funding received via Justice appropriations accounts as well as reimbursements from other Justice and federal agencies. First, OST receives funding via three appropriations accounts enacted in the appropriations law for the Justice Department. From these appropriations accounts, OJP allocates amounts to NIJ. The NIJ director suballocates part of the NIJ funds for OST programs. In addition, OST receives reimbursements from other Justice and federal agencies in exchange for OST’s management of specific projects of those agencies, such as ballistic imaging evaluation for the FBI. Table 1 lists NIJ allocations from the Justice appropriations accounts that go toward funding OST programs.

OST’s budgetary resources almost quadrupled from fiscal year 1995 to 1996, increased 70 percent from fiscal year 1999 to 2000, and increased 63 percent from fiscal year 2001 to 2002. While resources decreased 24 percent from fiscal year 2002 to 2003, OST’s fiscal year 2003 level still represents a 157 percent increase over the fiscal year 1999 level.

Certain Allocations Contributed to the Increase in Budgetary Resources since 1995

Our analysis of OST’s yearly budgetary resources from fiscal year 1995 to fiscal year 2003 showed that the overall increase can be attributed to the introduction of new NIJ allocations and large increases for existing ones. The NIJ allocations that contributed to the overall increase in OST’s budgetary resources are most notably the Crime Lab Improvement Program, DNA Backlog Reduction, Safe Schools Technology R&D, and Counterterrorism R&D allocations. Table 2 shows figures for all years in constant 2002 dollars.

All dollar figures used in this narrative are in constant 2002 dollars, except as noted otherwise.

Fiscal years 1995-1996: The $39.4 million (298 percent) increase from $13.2 million to $52.6 million primarily came from two NIJ allocations totaling $35.4 million.

Local Law Enforcement Block Grant (LLEBG) initiated with $22.2 million. Reimbursement of funds increased by $13.2 million (471 percent) from $2.8 million to $16.0 million.

Fiscal years 1999-2000: The $55.6 million (70 percent) increase from $79.5 million to $135.1 million primarily came from three NIJ allocations totaling $51.7 million.

DNA Backlog Reduction initiated with $15.6 million. Safe Schools Technology R&D allocation initiated with $15.6 million. Counterterrorism R&D increased by $20.5 million (193 percent) from $10.6 million to $31.1 million.

Fiscal years 2001-2002: The $103.4 million (63 percent) increase from $164.6 million to $268.0 million primarily came from three NIJ allocations totaling $95.3 million.

Reimbursement of funds increased by $55.6 million (209 percent) from $26.6 million to $82.2 million.

DNA Backlog Reduction increased by $24.3 million (227 percent) from $10.7 million to $35 million.

Crime Lab Improvement Program increased by $15.4 million (79 percent) from $19.6 million to $35 million.

To be consistent with the report narrative and to show trends, figures in table 2 are in constant 2002 dollars. A table with the figures in current dollars can be found in appendix II.

OST had a $63.8 million (24 percent) decrease in total budgetary resources from fiscal years 2002 to 2003, largely attributed to its not receiving fiscal year 2003 Counterterrorism R&D resources, which totaled $45.3 million in fiscal year 2002. According to OST, its counterterrorism resources were transferred to the Department of Homeland Security’s Office of Domestic Preparedness. There was also a $26.2 million decrease in the reimbursement of funds from other agencies. However, OST’s fiscal year 2003 level still represents a 157 percent increase from fiscal year 1999.

Range of OST’s Program Responsibilities Has Changed

The range of OST’s program responsibilities has changed over the years from primarily law enforcement and corrections to include broader public safety technology R&D. This has happened as more and more of OST’s budgetary resources were directed to be spent on specific recipients and projects. Appropriated funds, for example, are sometimes designated for specific recipients or projects in public law. In addition, guidance on the spending of appropriated funds may be provided through congressional committee reports. Of the more than $1 billion (in constant 2002 dollars) that OST programs received from fiscal years 1995 to 2003, $532.6 million, or 51 percent, was designated for specific recipients and projects in public law or subject to guidance in committee reports designating specific recipients or projects. Of the $532.6 million, $249.8 million was designated in public law for specific recipients or projects while $282.8 million was specified in committee report guidance for specific recipients or projects.

In addition to the $532.6 million designated in public law for specific recipients or projects or subject to guidance in committee reports for specific recipients or projects, another $217.1 million was reimbursements from other Justice and federal agencies in exchange for OST’s management of specific projects of those agencies. Thus, the total spending either directed for specific recipients and projects through public law, subject to committee report guidance designating specific recipients or projects, or received as reimbursements, amounts to $749.7 million, or 72 percent, of OST’s total budgetary resources.

The range of OST’s program responsibilities has changed to include such areas as school safety and counterterrorism. In fiscal year 1999, a Safe Schools Initiative program was established pursuant to conference committee report guidance with $10 million directing NIJ to develop school safety technologies. In another example, OST’s counterterrorism R&D program, initially funded by public law in fiscal year 1997, received $147.3 million through fiscal year 2002, $96.6 million of which was specified in conference report guidance for three recipients from fiscal years 2000 to 2002—Oklahoma City National Memorial Institute for the Prevention of Terrorism ($37.8 million), Dartmouth College’s Institute for Security Technology Studies ($51.8 million), and the New York University’s Center for Catastrophe Preparedness and Response ($7 million).

OST’s program responsibilities have also changed to expand the focus on investigative and forensic sciences. Our review of OST’s budgetary resources for fiscal years 1995 through 2003 shows that budgetary resources for investigative and forensic sciences equals at least $342.1 million in constant fiscal year 2002 dollars, or about one-third, of its $1 billion in budgetary resources, as shown in table 3. The proportion of investigative and forensic sciences annual funding to total OST funding rose from 6 percent ($800,000) in fiscal year 1995 to 52 percent ($106.0 million) in fiscal year 2003.

OST Delivers Three Groups of Products Through Various Methods

OST delivers many products, which we categorized into three groups, and uses various methods to deliver them. These three groups are (1) information dissemination and technical assistance; (2) the application, evaluation, and demonstration of existing and new technologies for field users; and (3) technology R&D. According to OST, as of April 2003, it had delivered 945 products since its inception. Furthermore, OST identified an additional 500 products expected from ongoing awards. Figure 2 shows our distribution of OST’s delivered products by group. We recognize, as OST officials told us, that the groups overlap and there is not a clean division between them. For example, while reports are associated with information dissemination, they may also result from the technology R&D group. OST has reviewed our classification of products and agrees that it is generally accurate. Because classification of some products is based on a judgment call, the proportions of products in each group should be considered approximations.

OST’s Range of Products

The following examples, while not exhaustive, indicate the wide range of OST’s products.

Reports on topics such as analysis of DNA typing data, linguistic methods for determining document authorship, a pepper spray projectile and disperser, and gunshot residue detection and interpretation.

Prototypes of products including ground-penetrating radar, ballistics matching using 3-dimensional images of bullets and cartridge cases, and an optical recognition system to identify and track stolen vehicles.

Evaluations of technology including prison telemedicine networks, police vehicles, and protective gear.

Guides on topics such as electronic crime scene investigation, use of security technologies in schools, and antennas for radio communications. For a more detailed description of OST’s products and further examples, see appendix III.

Information Dissemination and Technical Assistance

Information dissemination and technical assistance represents about 63 percent of OST’s delivered products. OST provides information to its customers in a variety of ways. For example, OST provides guidance to R&D laboratories on the needs of public safety practitioners. To public safety practitioners, OST recommends certain public safety practices, tools, and technologies. Through its Office of Law Enforcement Standards, OST develops performance standards to ensure that commercially available public safety equipment, such as handheld and walk-through metal detectors, meets minimum performance requirements. OST also helps its customers enhance their technical capacities by providing them with training and technical assistance through its Crime Lab Improvement Program (which also provides supplies and equipment), DNA Backlog Reduction Program, and network of technology centers. OST also uses the R&D expertise and experience of already established laboratories and other R&D organizations to provide additional guidance for managing specialized technology projects. Further, OST helps its customers receive surplus federal equipment by acting as their liaison to the equipment transfer program of the Department of Defense. For example, equipment transferred ranges from armored vehicles to boots and uniforms.

In addition, OST sponsors conferences, workshops, and forums that bring together its customers, technologists, and policymakers. For example, it sponsors the Mock Prison Riot, an annual event demonstrating emerging technologies in riot training scenarios held at the former West Virginia Penitentiary in Moundsville, West Virginia. This event brings together corrections officers and vendors for technology showcases and training exercises. Also, OST sponsors the Innovative Technologies for Community Corrections Annual Conference, among others.

Application, Evaluation, and Demonstration of New and Existing Technologies

Another OST product group is the application, evaluation, and demonstration of new and existing technologies, which represents about 20 percent of OST’s delivered products. Some of OST’s programs apply existing technology solutions in new ways to assist public safety operations. Examples of the application of new and existing technologies include developing methods for the collection and analysis of chemical trace evidence left from explosives and a handheld computer device provided to bomb technicians in order to access bomb data at the scene of incidents. In addition, OST tests commercially available products through NIJ-certified laboratories to determine whether they are in accordance with national performance standards. Examples of products evaluated against standards include body armor, handcuffs, and semiautomatic pistols. OST’s evaluations also include conducting field tests to compare different commercially available products of the same type to allow users to select the product that best suits their needs. OST also demonstrates technology resulting from R&D directly to its customers through OST- sponsored events. For example, the Critical Incident Response Technology Seminar, formerly known as the Operation America, demonstrates live-fire simulation for bomb technicians. The annual Mock Prison Riot demonstrates emerging technologies for use by corrections officers and tactical team members.

Technology R&D

About 17 percent of OST’s delivered products were related to technology R&D, which involves the development of prototype devices, among other efforts. According to OST, R&D in its early stages includes development of prototypes and demonstration that a principle can be proven. Applied R&D, which also involves the development of prototypes, includes technologies that are made available to public safety agencies, generally through OST-assisted commercialization. Examples of products resulting from OST’s applied R&D range from a bomb threat training simulator, facial recognition technology for internet-based gang tracking, to a personal alarm and location monitoring system for corrections officers.

According to OST, R&D in its early stages begins with testing technology concepts, exploring solutions, and deciding whether continued development is warranted. If OST decides to support product development and if it has available funds, it awards funding to develop, demonstrate, and evaluate an experimental prototype, which is then further developed into an initial engineering prototype, and then demonstrated and evaluated. If the prototype proves successful, OST demonstrates a “near commercial” model to its customers for their evaluation.

While OST does not directly commercialize the results of its technology R&D, it does provide prototypes to local users for field-testing and assists in linking prototypes with potential commercial developers. OST officials believe it would be a conflict of interest and therefore inappropriate for them to promote one vendor or technology over another or try to dictate what equipment their customers should purchase. OST’s role in commercialization is to bring technologies and potential manufacturers together so that the manufacturers can determine the feasibility of commercializing the technologies.

OST’s Methods for Delivering its Products

OST delivers its products to its customers through a variety of methods. (We recognize that products are sometimes delivery methods. For example, a publication can be both a product resulting from research and a method of information dissemination.) Besides publications, OST’s methods for delivering information and technical assistance include mass mailings; downloadable material from its Web site; panels, boards, and working groups; training, support, and presentations; and programs to enhance the capacity of public safety agencies.

OST also delivers its products related to application, evaluation, and demonstration through various means. For example, private industry provides new and existing technologies to OST; in turn, OST informs its customers of the results of using these technologies in new ways. OST publishes user guides and the test results of its evaluations of commercially available equipment (both standards-based and comparison- based). Seeking to further educate its customers, OST demonstrates new technology at technology fairs, providing “hands on” opportunities to use it.

For its R&D products, OST may test “near commercial” prototypes in particular settings. For example, OST may install in a police agency a prototype technology that facilitates communications among public safety agencies and across jurisdictions. If the technology is effective, the police agency may incorporate the technology directly into its operations, before the technology has become a commercial product.

OST’s Performance Measurement Efforts Do Not Fully Meet Requirements

OST’s efforts to measure its performance results, including the usefulness and effectiveness of its products, do not fully meet applicable requirements. To help Justice comply with the Government Performance and Results Act of 1993 (GPRA), OST establishes goals and develops performance measures to track its progress. GPRA, which mandates performance measurements by federal agencies, requires, among other things, that each agency measure or assess relevant outputs and outcomes of each program activity. According to GPRA, the Office of Management and Budget (OMB), and GAO, outcomes assess actual results as compared with the intended results or consequences that occur from carrying out a program or activity. Outputs count the goods and services produced by a program or organization. Intermediate measures can be used to show progress to achieving intended results. Subsequent OMB and committee report guidance on GPRA, and previous GAO reports recognize that output measures can provide important information in managing programs. However, committee report guidance emphasizes using outcome measures to aid policy makers because such measures are key to assessing intended results.

OST Performance Measures Do Not Measure Results

The performance measures that OST has developed do not measure results. According to the NIJ director, the Assistant Attorney General (AAG) in April 2002 issued a memorandum requiring NIJ, including OST, to develop outcome measures for fiscal year 2004. In August 2002, the NIJ Director responded by stating that OST had indeed developed outcome measures for its programs. In its fiscal year 2004 performance plan, OST established goals for 11 of its initiatives and developed 42 measures for assessing the achievement of those goals. However, based on our review of OST’s performance plan, OMB guidance on GPRA, and GAO definitions of outcome, output, and intermediate measures, we determined that of the 42 measures, none were outcome-oriented, 28 were output-oriented, and 14 were intermediate. See table 4 for GAO’s determination of the measures and appendix VI for further details of our results.

According to Justice officials, R&D activities present measurement challenges because outcomes are difficult or costly to measure. As the NIJ Director pointed out, a May 2002, White House OMB and Office of Science and Technology Policy memorandum concluded that agencies should not have the same expectations for measuring the results of basic R&D as they do for applied R&D. According to NIJ, relatively little of OST’s work is basic R&D. As shown earlier, most of OST’s products are related to information dissemination and technical assistance and the application, evaluation, and demonstration of existing and new technologies for field users.

We recognize that OST’s task in relation to measuring the results of even non-basic research is complex in part because of the wide array of activities it sponsors, and because of inherently difficult measurement challenges involved in assessing the types of programs it undertakes. For example, programs that are intended to deter crime face measurement issues in assessing the extent to which something (crime) does not happen. Nevertheless, improvement in measurement of program results is important to help OST ensure it is doing all that is possible to achieve its goals. It is worth noting that an outcome measure in relation to one OST program was discussed by the NIJ Director in a May 2002 statement to Congress. In this statement, the Director provided an example of an outcome from the Convicted Offender DNA Backlog Reduction Program. The Director stated that as a direct result of the program, approximately 400,000 convicted offender samples and almost 11,000 cases with no suspect were analyzed. According to the NIJ Director, as of May 14, 2002, more than 900 “hits” had been made on the FBI’s Combined DNA Index System (CODIS) database as a direct result of the program, that is, 900 cases previously unsolved had been reopened. This information indicates how the program is achieving its intended results in addressing unsolved cases. Although this example seems to be a credible outcome measure, it is not included in OST’s fiscal year 2004 performance plans.

Limitations in OST’s Efforts to Measure Effectiveness of Information Dissemination

OST efforts to measure information dissemination effectiveness have been limited. One of the purposes of GPRA is to improve federal program effectiveness and public accountability by promoting a new focus on results, service quality, and customer satisfaction. Surveys to gauge customer satisfaction represent one step toward finding out whether customers have received information and whether they deem it of value. However, these surveys have limitations in determining the extent to which the information has been acted upon and resulted in intended improvements. Thus, surveys such as these are more likely to be intermediate measures (Did information get transferred?) than outcome measures (Did information get transferred, acted upon, and achieve a result?).

In 1998, NIJ initiated an effort to report the results of surveys to measure the satisfaction of participants at all conferences, workshops, and seminar series. OST reported on the “grantee level of satisfaction with NIJ conferences” for fiscal years 1998-2000. However, in the fiscal years 2001- 2004 GPRA performance plans, OST discontinued tracking the surveys because OJP and NIJ had ceased tracking these data as a performance measure.

In fiscal year 2001, OST attempted to evaluate the effectiveness and value of its TECHbeat newsletter. The survey sample of 5,500 was taken from a distribution of major readership groups on TECHbeat’s mailing list of 20,674. According to OST, the response rate for the survey was too low to produce statistically valid results: only 124 completed or substantially completed responses were collected. The surveyors also experienced a very low return on follow-up phone queries. According to the study, the primary reason for the exceedingly low response rate was that so many individuals on the mailing list had either changed jobs or were completely unfamiliar with TECHbeat. Given these results, OST is trying to improve the management and distribution of TECHbeat.

In fiscal year 2001, OST attempted to launch another effort to measure program results, service quality, and customer satisfaction, but funding for the effort was not provided. OST requested funding for an evaluation to measure the success of its outreach efforts, including those by its technology centers. The evaluation was to determine customer satisfaction with its strategies for outreach and communication and with its products. Specifically, OST planned to measure user satisfaction of the content, format, and delivery mechanisms of its efforts, such as technology information and assistance.

Most Studies of Other OST Initiatives Have Focused Primarily on Process

In fiscal years 1998 and 1999, OST funded eight outside studies of some of its science and technology initiatives (see table 5). Our review of these studies showed that seven of the eight studies focused on management and organizational processes, and one was outcome-oriented. Management and process evaluations can be useful tools for examining how a program is operating and can offer insights into best practices. They do not assess whether a program is achieving its intended results.

Efforts Are Under Way to Address Performance Measurement of Technology Centers

The Homeland Security Act of 2002 requires NIJ to transmit to Congress by late November 2003 a report assessing the effectiveness of OST’s existing system of technology centers and to identify the number of centers necessary to meet the technology needs of federal, state, and local law enforcement in the United States. According to NIJ, in response to the Homeland Security Act requirement, it has initiated a study to assess the impact and effectiveness of the technology center system and how it can be enhanced to meet the evolving science and technology research and technology needs of the state and local public safety community. NIJ also stated that the report would address the functions that the technology center system must provide to transfer NIJ’s research and development results to practice in the criminal justice system. NIJ and OST have failed to provide us with information detailing the methodology of the study, so we cannot comment on the likelihood that this study will produce the information sought by Congress. Additionally, according to OJP, the technology centers are in the process of developing outcome measures to demonstrate the impact of their activities.

According to NIJ, OJP has implemented additional performance measures developed in May 2003 that will apply to NIJ, including OST. However, OJP said it would defer implementing the measures related to the technology centers until the results of the technology center study are known and NIJ has a chance to take action, if warranted.

Measuring Results Is Difficult but Feasible

We acknowledge that measuring results using outcome measures is difficult, and may be especially so in relation to some of the types of activities undertaken by OST. Indeed, given the types and wide range of program goals for OST efforts—solving old crimes, saving lives, and reducing property loss—it may be the case that for some programs intermediate measures represent the best feasible measure of results. We note that approximately 63 percent of OST’s products fall into the category of information dissemination and technical assistance, aimed at informing customers and ultimately encouraging adoption of research results that lead to increased efficiency and effectiveness. There are strategies available that have been used by other federal agencies to take steps toward assessing the effectiveness of information dissemination and technical assistance efforts. For example, a recent GAO report outlines various strategies to assess media campaigns and informational seminars, including immediate post workshop surveys and follow-up surveys and the use of logic models to define measures of a program’s progress toward intended results and long-term goals.

Conclusions

Given the wide range of its products, OST has the potential to significantly improve the technological capabilities of federal, state, and local public safety agencies. However, the lack of information about the results of program efforts, or the assessment of progress toward goals, means that little is known about their effectiveness. While developing outcome measurements in many research-related programs is extremely difficult, there are various performance measurement strategies that other federal programs have used for assessing information dissemination, technical assistance and other R&D activities that might be applied to OST’s programs. It is important to develop outcome measurements where feasible, or intermediate measurements where appropriate, to assist Congress, OST and NIJ management, and OST’s customers to better assess whether investment in OST’s programs is paying off with improved law enforcement and public safety technology.

Recommendation

To help ensure that OST does all that is possible to measure its progress in achieving goals through outcome-oriented measures, we recommend that the Attorney General instruct the Director of NIJ to reassess the measures OST uses to evaluate its progress toward achieving its goals and to better focus on outcome measures to assess results where possible. In those cases where measuring outcome is, after careful consideration, deemed infeasible, we recommend developing appropriate intermediate measures that will help to discern program effectiveness.

Agency Comments and Our Evaluation

We provided a copy of a draft of this report to the Attorney General of the United States for review and comment. In an October 30, 2003, letter, the Assistant Attorney General (AAG) for OJP commented on the draft. Her comments are summarized below and presented in their entirety in appendix VII. OJP also provided technical comments, which have been incorporated into this report where appropriate.

In the AAG’s written response, the Justice Department concurred with our recommendation that NIJ reassess the measures OST uses to assess program outcomes. In response to our recommendation, the AAG reported that she has directed the NIJ Director, to reassess NIJ’s performance measures for OST and to refine them, where possible, in order to focus them more toward measuring outcomes.

While the AAG agreed with our recommendation, she also made several other comments. First, she commented that developing numerical outcome measures like those urged by GAO is a particular challenge for R&D activities. As stated in our report, we recognize that measuring results using outcome measures is difficult and may be especially so in relation to some of the types of activities undertaken by OST. Our reference to a numerical measure is meant only as an example of how one of OST’s program activities can be linked to intended results. We believe that further consideration of measures, both quantitative and qualitative, could improve the assessment of results for R&D as well as other OST programs. Our report also notes that relatively little of OST’s work is R&D. The majority of OST’s products are in the category of information dissemination and technical assistance.

Second, the AAG noted that GAO did not reach any conclusions in its discussion of OST’s growth in budgetary resources, changes in program responsibilities, management of programs, and delivery of its products. The AAG noted that Justice believed that OST’s record is outstanding. Neither OST nor we can determine whether OST’s efforts in these areas are successful or otherwise, given that OST has not developed measures to assess their outcomes. Therefore, it is not possible to draw conclusions.

Third, the AAG indicated that GAO did not discuss in detail that over one- half of OST’s funds were designated by Congress for specific recipients and projects. She noted that GAO missed an opportunity to inform the requester of the impact of Congress’ recent decisions regarding OST. We reported that 51 percent of OST’s budgetary resources were designated for specific recipients and projects in public law or subject to guidance in committee reports.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 10 days from its issue date. At that time, we will send copies of the report to the Attorney General, appropriate congressional committees and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. Major contributors to this report are listed in appendix VIII.

If you or your staff have any questions concerning this report, contact me on (202) 512-8777.

Appendix I: Scope and Methodology

To answer our objectives, we interviewed National Institute of Justice (NIJ) and Office of Science and Technology (OST) officials and collected documents at OST’s office in Washington, D.C., and at three of OST’s technology centers—the National center in Rockville, Maryland; West center in El Segundo, California; and Border Research and Technology Center in San Diego, California. We selected the Rockville center because of its proximity to Washington, D.C., and the other two centers because of their locations and particular areas of technology and technical concentrations. We also interviewed a small group of OST’s customers— federal, state, and local law enforcement, and corrections and public safety officials—who were selected by officials at the El Segundo and San Diego centers. In addition, we analyzed information that is available on the National Institute of Justice’s public Web site.

To determine OST’s budgetary resources and amounts from fiscal year 1995 to fiscal year 2003 and the changes in OST program responsibilities, we reviewed NIJ and OST budget documents, interviewed officials in OST’s Technology Management and the OJP’s Office of Budget and Management Services, and reviewed pertinent appropriations laws and committee reports covering that period. To determine the amount of OST budgetary resources that were directed to specific recipients and projects, we compared OST’s budget documents that listed individual recipients and projects with the public laws and reports. We defined directed spending as spending for specific recipients and projects designated in appropriations laws or subject to congressional committee report guidance designating specific recipients or projects. We did not determine the amount of reimbursable projects designated in public laws or specified in committee reports because those projects were not originally allocated to OST. Instead, we considered all the reimbursable projects to be specific projects for which OST was directed pursuant to its agreements with other agencies on spending its reimbursable funds.

To determine the changes in OST’s program responsibilities, we analyzed the year-to-year changes in its budget and program scope. To determine the amount of OST’s budgetary resources used for investigative and forensic sciences for fiscal years 1995-2003, we compared OST’s portfolio description and NIJ’s definition of forensic sciences with the individual budget program and project items listed in OST’s budget documents for each fiscal year. However, while we recognize that OST’s technology centers and their technical partners include investigative and forensic sciences in their provision of technical assistance, we did not attempt to determine the amount of center funds associated with investigative and forensic sciences because the budget documents we received from OST did not break out such amounts within the funding awarded to the centers. Therefore, our determination that $342.1 million of OST’s total funding supported investigative and forensic sciences did not include such amounts.

To determine the amounts of funding awarded to the technology centers, we analyzed databases on all of the products OST has produced through April 2003 and the associated grants, interagency agreements, and cooperative agreements and their amounts.

To determine the composition of OST’s products and how OST delivers the products to its customers, we analyzed OST documents and a database of all the products associated with its past and ongoing awards, from inception through April 2003, that were delivered or anticipated to be delivered. While the database included the award amounts associated with the products, it was not possible to reliably associate the award amounts for each product or type of product because multiple types of products could result from individual awards. We also conducted interviews with the parties mentioned above.

For the budget and product data that OST provided us, we assessed the reliability of these data by examining relevant fields for missing data, conducting range checks to identify any potentially erroneous outliers and inspecting a subset of selected data elements that were common to two or more data sets. In addition, we independently verified selected budget data back to appropriations legislation and Committee reports. In conducting our analyses, we identified some potential data errors or reliability problems. When this occurred, we contacted agency officials to address and resolve these matters. However, we did not verify the budget or product data back to source materials. Overall, we determined that budget or product data provided to us is adequate for the descriptive purposes it is used in this report.

We examined OST’s efforts to measure performance by interviewing officials on this matter at OJP, NIJ, and OST in the Washington, D.C., office along with officials and staff at the technology centers, and current and previous Advisory Council officials. We also reviewed related agency documents, such as the OJP mission statement and performance plans; NIJ strategic planning documents and website pages, annual performance plans and performance reports, and GPRA documents; policies and procedures, Department of Justice memoranda, OST internal planning and reporting documents, program descriptions and documentation, and other related documents.

As part of our examination, we reviewed OST’s fiscal year 1997 to 2004 goals and measures as presented in OST’s GPRA performance plans. We focused our review on OST’s fiscal year 2004 performance plan and measures. As part of our review of these goals and measures, we made a determination as to whether the performance measure was output, outcome, or intermediate-oriented. To make this determination about the types of performance measures contained in OST’s performance plans, we compared the measures used in the plans with the requirements of GPRA, accompanying committee report, OMB’s guidance on performance measurement challenges (Circular A-11), Justice’s guidance to its components for preparing performance measures, and previous GAO work on GPRA.

Also included in our examination of OST performance measurement efforts were studies prepared by external parties under grants from OST that reviewed selected OST initiatives such as portfolio areas, projects, and programs. In response to our request for all of OST’s efforts to assess its programs, OST provided eight outside studies funded from fiscal years 1998 to 1999. For example, the Pymatuning Group, Inc., conducted an “Assessment of the National Law Enforcement and Corrections Technology Center (NLECTC) Program,” which described the operations of the OST’s regional technology centers network. We reviewed all eight of the outside studies for performance information on the OST initiatives being examined in the report. We examined the studies to determine whether they provided information that would be considered consistent with an outcome-oriented evaluation as defined by our criteria.

The scope of this review was limited to OST, and therefore we cannot compare OST’s efforts to measure the performance of its programs or the amount of funding directed to specific recipients and projects with the efforts and funding of any other federal R&D agencies. We performed our audit work from September 2002 to September 2003 in Washington, D.C., and other cited locations in accordance with generally accepted government auditing standards.

Appendix II: Bugetary Resources for OST’s Programs in Current Year Dollars

Reimbursements from other Justice and federal agencies

Appendix III: OST’s 10 Categories of Products

While we divided OST’s products into three groups for our reporting purposes, OST divides them into 10 categories. (See table 7 for GAO’s 3 groupings of OST’s 10 categories.) In regrouping OST’s 10 categories, we recognized, as OST officials told us, that the 10 categories overlap and there is not a clean division between them. We also recognized that many of OST’s products could also be considered a delivery method. For example, publications, such as the TECHbeat newsletter, are OST products that can also represent a method of delivery for OST technology information. OST has reviewed our classification of products and agrees that it is generally accurate.

Appendix IV: OST’s Portfolio Areas

OST has organized its individual projects to develop, improve, and implement technology for public safety agencies into nine portfolio areas. As of April 2003, these portfolio areas included critical incident technology, for first responders and investigators protecting the public in the event of critical incidents such as natural disasters, industrial accidents, or terrorist acts; communications interoperability and information sharing, enhancing communication among public safety agencies through wired links, wireless radios, and information networks, even when disparate systems are involved; electronic crime, supporting computer forensic laboratories, publishing guides for handling electronic evidence, and developing computer forensic tools; investigative and forensic sciences, funding at the state and local levels for DNA-typing of convicted offenders and use of DNA-typing in the investigation of unsolved cases, and developing tools for forensic casework; crime prevention technologies, including contraband detectors, sensors and surveillance systems, and biometric technologies; protective systems technologies, including body armor; “smart” handguns, which fire only upon recognition of, for example, a certain handprint or password; puncture resistant gloves; better handcuffs; better concealed weapon detection; and personnel tracking and location technologies; less-than-lethal technologies, developing alternatives to lethal force, including technologies involving electrical or chemical effects, light barriers, vehicle stopping, and blunt trauma, and evaluating and modeling the effects of these technologies; learning technologies, developing technology tools for agencies to use in training their personnel, including use of the internet, CD-ROMs, and video-based and interactive simulations; and standards and testing, ensuring that the equipment public safety agencies buy is safe, dependable, and effective.

Appendix V: OST’s Operations

As with other federal agencies, OST’s operations involve multiple levels of internal organization and multiple kinds of external partners. OST’s multiple levels of organization include a Washington, D.C., office that manages its technology programs and a network of technology centers around the country that provide technical assistance to OST’s regional customers. OST also collaborates with other R&D entities, such as those in the Departments of Defense and Energy and public and private laboratories, by forming technical partnerships in order to leverage already established technical expertise and resources to support their program efforts. Another aspect of OST’s complex operations is the need to determine OST’s own priorities and the priorities of its customers, which involves Washington and regional center staff collaborating formally and informally with a myriad of federal, state, and local officials, as well as with one another.

OST Has Multiple Levels of Organization

OST’s multiple levels of organization include a Washington, D.C., office and technology centers, as well as technical partnerships with government, public and private R&D and public safety organizations. As of September 2003, OST’s Washington office consisted of 25 full-time- equivalent Justice staff divided into three divisions and under the Assistant NIJ Director for OST. Responsibility for managing these programs is divided among the three divisions. (See figure 3 for OST’s organizational structure.)

Research and Technology Development Division manages electronic crime (including cybercrime), critical incidents and counterterrorism, communications interoperability and information sharing, crime prevention, learning technology tools, less-than-lethal technologies, standards development, school safety, and corrections technologies.

Investigative and Forensic Sciences Division manages DNA-related R&D and other investigative and forensic sciences, such as fingerprint analysis, and includes the Crime Laboratory Improvement Program projects, DNA Backlog Reduction projects, and DNA research and development projects.

Technology Assistance Division, through the technology center network, provides training and technical advice to, and identifies technologies for, OST’s customers, and oversees OST’s network of 10 technology centers (see figure 4). The technology centers are another source of technical advice for OST’s customers.

OST’s Technology Centers

OST’s network of 10 technology centers provides technical assistance, among other things, to OST’s customers. From fiscal year 1995 to fiscal year 2003 (as of July 2003), funding support for the centers totaled $171.7 million. (See table 8 for funding by center.) The technology centers comprise six regional centers and four specialty centers. While the regional centers assist OST’s customers by region—Northwest, West, Rocky Mountain, Northeast, Southeast, and National—they are expected to coordinate and collaborate among one another regardless of where the resources and capabilities are located. Each of these 6 centers works with a regional advisory council comprising state and local law enforcement, corrections, and public safety representatives.

As described below, the four specialty centers provide specialized expertise and services.

The Office of Law Enforcement Standards tests commercially available equipment and develops minimum performance standards for such equipment.

The Office of Law Enforcement Technology Commercialization, Inc., assists inventors and developers, among others, in commercializing technologies.

The Border Research and Technology Center aids in the development of technologies for agencies concerned with law enforcement at the northern and southern borders.

The Rural Law Enforcement Technology Center aids rural and small- community law enforcement and corrections agencies.

OST’s Technical Partnerships for Long- Term Support

In addition to forming divisions and technology centers, OST has also formed partnerships with governmental, public and private R&D organizations, agencies, and working groups. According to OST officials, an advantage of these partnerships is that OST can leverage the expertise and resources of already established R&D facilities without having to create their own. These partners have included corporations, such as Georgia Tech Research Corporation and L-3 Communications, Analytics Corporation; state and local agencies, such as the Houston Police Department and the Washington Metropolitan Area Transit Authority; academic institutions, such as the University of Virginia and Syracuse other federal government agencies, such as the Department of Defense’s Army Training and Doctrine Command, and the Department of Transportation’s Federal Aviation Administration; and foreign government organizations, such as the Royal Canadian Mounted Police, the United Kingdom Police Scientific Development Branch, and the government of Israel.

Each of OST’s technology centers is affiliated with one or more of OST’s technical partners. These technical partners are awarded funding in exchange for providing staff and facilities to the technology centers. Table 9 lists OST’s partners and their affiliations, and funding they received to support the centers through June of fiscal year 2003.

OST Collaborates with Many Customers and Partners to Determine Program Priorities

To determine its program priorities, OST collaborates with its many customers and partners. Staff at both OST’s Washington, D.C., office and its technology centers are involved in helping OST to set program priorities. The staff report the results of their collaboration through formal meetings, periodic reports, and informal communication. Input is exchanged continually between OST’s customers and staff and within its multiple levels of organization. Using their input, the NIJ Director determines OST’s program priorities. (See figure 5 for the stakeholders, partners, and customers that contribute to the setting of OST’s priorities.)

OST Collaborates with Government Agencies, Research and Professional Communities, and Centers

OST’s three divisions collaborate with other U.S. government agencies, the research and professional communities, and its technology centers to solicit input for setting priorities. Also, the divisions work with public safety practitioners at the state and local levels by, for example, meeting with grantees and assessing their needs.

The Investigative and Forensic Sciences Division collaborates with, and receives input from, researchers, academia, and the forensic laboratory community to help set program priorities. It also collaborates with, for example, the FBI and the interagency Technical Support Working Group.

The Research and Technology Development Division receives input through its collaboration with other federal agencies, such as the FBI, Drug Enforcement Administration, U.S. Secret Service, and White House Office of National Drug Control Policy. The division also participates in interagency working groups, such as for school safety and the Technical Support Working Group. Through these collaborations, OST can develop and share technologies used by both its customers and other agencies. For example, OST works with the Department of Defense to conduct less-than- lethal weapons R&D for law enforcement.

The Technology Assistance Division is primarily responsible for receiving input from OST’s technology centers. The centers solicit input from customers through their outreach efforts, such as technical assistance, e-mail exchanges, and telephone calls. The centers are also required to use OST’s web-based reporting system to record information on their customers’ requests for technical assistance. The centers are also required to submit monthly reports on their activities and finances.

Advisory Councils and Federal, State, and Local Public Safety Agencies Collaborate with OST’s Technology Centers

OST’s technology centers solicit input from the national and regional advisory councils that OST created to determine and advocate for the particular needs of its customers. Members of the national advisory council are selected by the technology centers and represent federal, state, and local public safety agencies, as well as international criminal justice organizations. Among its duties, this national advisory council identifies the present and future equipment and technology needs of OST’s customers and reviews the programs of the technology centers. In addition, the national advisory council recommends (1) ways to improve the technology centers’ programs’ relevance to the needs of the centers’ customers and (2) broad priorities for the technology center network and OST that are consistent with the needs of their customers.

Each technology center has a regional advisory council. The regional advisory councils consist of a cross-section of law enforcement and other public safety officials who represent the interests of state and local officials. The regional advisory councils solicit input from the state and local agencies serviced in their regions, advise and support their respective center directors on their customers’ problems and needs, and advocate for resource support and improvements required by their customers. Through this method of sharing information, OST can better understand the needs of its customers. For example, OST’s regional councils can represent the unique needs of their customers that the national advisory council or the technology centers might not be aware of.

Appendix VI: OST’s Goals in its Fiscal Year 2004 Performance Plan and GAO’s Assessment

Reduce DNA backlog and support a functioning, active system, which can solve old crimes and prevent new ones from occurring.

1. Number of labs demonstrating improved access to external capabilities and increased lab capabilities. 2. Number of samples (1) analyzed using the selected DNA markers that are required by the FBI’s national Combined DNA Index System (CODIS) database, and (2) made available for CODIS. 3. Number of states that have experienced an increase in the number of samples they have contributed to the national database.

Reduce DNA backlog and support a functioning, active system, which can solve old crimes and prevent new ones from occurring.

4. Number of DNA samples from cases where there is no known suspect.

Improve quality, timeliness, and credibility of forensic science services.

5. Number of forensic labs with improved analytic and technological resources.

Improve the ability of public safety responders, including law enforcement and corrections officers, to deal with critical incidents, save lives, and reduce property loss.

6. Number of technology demonstrations and test indicators that describe the goods and services produced. 7. Number of prototype technologies developed. 8. Number of guides, standards, and assessments in progress. 9. Number of guides, standards, and assessments completed. 10. Number of technologies introduced in law enforcement and corrections agencies.

Develop faster and more powerful tools and techniques for the analysis of DNA evidence. These new tools and techniques will result in more crimes prevented and solved and more perpetrators brought to justice.

11. Number of projects researching new forensic DNA markers. 12. Number of development/validation studies for forensic DNA techniques. 13. Number of computer programs developed for forensic DNA analysis. 14. Number of prototypes and tools for forensic DNA analysis.

Assist in applying technology to reduce the vulnerability of critical infrastructure; detect weapons and other contraband; improve technologies to locate and differentiate between individuals in structures; leverage information technology to enhance the responder community’s ability to anticipate and deal with critical incidents; identify and respond to terrorist attacks involving chemical, biological, and other unconventional weapons; and develop needed standards.

15. Number of technology demonstrations and tests. 16. Number of prototype technologies developed. 17. Number of guides, standards, and assessments in progress. 18. Number of guides, standards, and assessments completed. 19. Number of technologies introduced in law enforcement and corrections agencies. 20. Number of technology demonstrations.

Assist school administrators and law enforcement in creating a safer and more productive learning environment. Safe, effective, appropriate, and affordable technologies can affect the perception and reality of safe schools.

21. Number of conferences and forums. 22. Number of school safety technology products.

Provide immediate results in solving more crimes, bringing to justice more criminals, and improving administration of justice through the presentation of strong, reliable forensic evidence at trial. forensic services. 24. Number of capacity-building forensic R&D and validation projects funded. 25. Number of forensic technology training tools developed and distributed. 26. Number of labs providing continuing education or advanced training to crime analysts. 27. Number of crime labs with increased capacity for implementation of new forensic capabilities (including DNA analysis). 28. Number of capacity-building forensic R&D and validation projects completed and impacting crime labs. 29. Number of labs establishing new forensic capabilities. 30. Number of labs expanding current forensic capabilities. 31. Number of labs experiencing a reduction in time needed for evidence analysis. 32. Number of labs experiencing a reduction in backlogged evidentiary sample analysis.

Help the public safety community make informed decisions about products being marketed for public safety personnel.

33. Number of methods for examining evidentiary materials developed. 34. Number of standards for equipment and operating procedures developed. 35. Law enforcement technology deliverables (standards, product performance evaluations, product guides).

Develop a firearm that could save the lives of law enforcement officers and members of the public that they encounter while performing their duties.

36. Successful demonstration of prototype recognition system for smart gun. 37. Failure mode analysis for prototype recognition system for smart gun. 38. Incorporation and demonstration of recognition system into firearm (where applicable). 39. Identification of appropriate biometric solutions for recognition system (where applicable).

Help state and local law enforcement, corrections, and public safety personnel do their jobs more safely and efficiently, thereby leading to greater administrative efficiencies, more crimes solved, and more lives saved.

40. Number of technology information documents distributed. 41. Number of practitioners trained through the Crime Mapping Program. 42. Savings to criminal justice agencies through the DOD’s Section 1033 Military Surplus Program. Section 1033 of the National Defense Authorization Act for Fiscal Year 1997 authorizes DOD to transfer excess military property to federal and state agencies to support law enforcement activities including counterdrug and counterterrorism activities.

Appendix VII: Comments from the Department of Justice

Appendix VIII: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, the following individuals contributed to this report: Samuel L. Hinojosa, Debra L. Picozzi, Katherine M. Davis, Richard Hung, Geoffrey R. Hamilton, Denise M. Fantone, Kristeen McLain, Elizabeth H. Curda, Rebecka Derr, Thomas M. Beall, and Leo M. Barbour.