Department of Justice: Status of Achieving Key Outcomes and Addressing Major Management Challenges
Released: 2001-07-26T00:00:00

Letter

Background

GPRA is intended to shift the focus of government decisionmaking, management, and accountability from activities and processes to the results and outcomes achieved by federal programs. New and valuable information on the plans, goals, and strategies of federal agencies has been provided since federal agencies began implementing GPRA. Under GPRA, annual performance plans are to clearly inform the Congress and the public of (1) the annual performance goals for agencies’ major programs and activities, (2) the measures that will be used to gauge performance, (3) the strategies and resources required to achieve the performance goals, and (4) the procedures that will be used to verify and validate performance information. These annual plans, issued soon after the transmittal of the president’s budget, provide a direct linkage between an agency’s longer-term goals and mission and day-to-day activities. Annual performance reports are to report on the degree to which performance goals were met. The issuance of the agencies’ performance reports, due by March 31 each year, represents a new and potentially more substantive phase in the implementation of GPRA—the opportunity to assess federal agencies’ actual performance for the prior fiscal year and to consider what steps are needed to improve performance and reduce costs in the future.

As the nation’s chief law enforcement agency, Justice is charged with, among other things, enforcing laws in the public interest and playing a key role in protecting the public from violence and criminal activity, such as drug smuggling and acts of terrorism. With a fiscal year 2001 budget of over $24 billion and a staff of about 111,000, including attorneys, investigators, and agents, Justice is a multifaceted organization whose functions range from securing the nation’s borders to helping state and local agencies improve their capacity to prevent and control crime. Justice’s responsibilities are divided among a number of major components, including the Drug Enforcement Administration (DEA), the Federal Bureau of Investigation (FBI), INS, the Office of Justice Programs (OJP), and the United States Marshals Service (USMS).

Assessment of Justice’s Progress and Strategies in Achieving Selected Key Outcomes

This section discusses our analysis of Justice’s performance in achieving the selected key outcomes and the strategies it has in place, particularly strategic human capital management and information technology, when appropriate, for accomplishing these outcomes. In discussing these outcomes, we have also provided information drawn from our prior work on the extent to which the agency provided assurance that the performance information it is reporting is credible.

Drug- and Gang-Related Violence

Overall progress made by Justice toward achieving less drug- and gang- related violence is difficult to ascertain because (1) three of nine performance measures did not have fiscal year 2000 targets to measure success and (2) Justice fell short of achieving its performance targets for four measures. Justice did not set fiscal year 2000 performance targets for its performance related to dismantling Asian criminal enterprises, dismantling Eurasian criminal enterprises, and cases in Indian Country.Justice did not set performance targets for these measures because it considered two of the measures to be new measures, and for the number of cases in Indian Country, Justice did not want to set performance levels because it believes that setting performance targets could cause the public to perceive law enforcement as engaging in “bounty hunting” or pursuing arbitrary targets merely for the sake of meeting particular goals. In addition, even though Justice indicated that the performance measures for dismantling Asian and Eurasian criminal enterprises were new measures, these same measures were included in Justice’s fiscal year 1999 performance report (albeit Eurasian was called Russian then).

Justice fell short of achieving the performance targets for four measures. For example, although close, Justice did not meet its performance target to perform 4.81 million criminal background checks. Justice reported that it had perfomed 4.49 million criminal background checks. Also, Justice did not meet its performance target to prevent 140,244 persons with criminal backgrounds from purchasing firearms. Justice reported it had prevented 71,890 ineligible persons from purchasing firearms. In its explanation of why it did not meet these performance targets, Justice noted that the reported targets for these measures were based on the assumption that all states and territories would be full participants in the National Instant Criminal Background Check System (NICS) program. Justice reported that only 27 states have become full participants since NICS began in November 1998. In an April 2000 report, we reported that the states generally are better positioned than the FBI to conduct background checks and that there are potential barriers to states participating in NICS. Justice’s performance report does not articulate the implications of potential barriers if the intent is for all states to participate.

Although close, Justice also did not meet its performance measure to initiate 20,000 new Interpol cases or enter into 10 new mutual legal assistance treaties with other countries. Justice reported that in fiscal year 2000, it had initiated 19,549 new Interpol cases and entered into 8 treaties. Justice noted that it had not met its performance targets for these measures because there are a number of outside entities (such as, state liaison offices, foreign governments, the State Department, the White House, and the U.S. Senate) that have key roles in these efforts and that Justice has limited control over actions of these entities. However, Justice did not discuss any actions that it might take to mitigate the effects of external factors. Our February 1999 report also suggested that to improve the usefulness of annual plans, agencies show how strategies will be used to achieve goals that include describing approaches to leverage or mitigate the effects of external factors on the accomplishment of performance goals.

For each performance measure, Justice included a brief explanation about data collection and storage, data validation and verification, and any known data limitations. There were data limitations associated with four of the databases used with the performance measures. Two of the data limitations appear to be more of a clarification rather than a limitation. The other two data limitation explanations were more significant. One limitation noted that a significant number of criminal history records were not complete and that state and local agencies and the courts needed to update and complete the records in a more timely manner. The performance report did not indicate actions or steps that might be taken to mitigate the data limitations with state and local agencies and the courts in order to improve the data’s reliability. For the other limitation, Justice reported that the current reporting system for the number of Interpol cases was severely limited. However, Justice reported that in fiscal year 2001 the database and procedures were to be validated for accuracy and redesigned for efficiency and that a comprehensive and flexible reporting system to extract the statistics from the database was to be developed.

Justice’s strategies and initiatives to achieve less drug- and gang-related violence generally seem reasonable and clear. However, Justice could improve its performance strategies by exploring potential coordination efforts that might be used to mitigate external factors and by considering the use of performance evaluations to better assess its progress toward achieving the outcome. For example, on the basis of its fiscal year 2000 performance, Justice modified its fiscal year 2001 performance target from 5.05 million to 4.54 million for the number of criminal background checks performed. Modifying the performance target in the short term is a reasonable step; however, Justice may want to discuss what it has considered in response to only 27 states participating in the NICS since it began in November 1998 and whether other strategies to assist states are needed to achieve the outcome. As mentioned earlier, Justice did not have performance targets for one measure because of concerns about pursuing targets merely for the sake of meeting a goal. We suggest that Justice could compare the relative effectiveness of programs using a program evaluation approach to provide an indication of its progress towards achieving its goals. Specifically, Justice has a goal to provide enforcement assistance and training to tribal governments to combat and reduce the incidence of violent crime on Indian Reservations, especially crime related to gang activity. This is a performance measure for which Justice did not provide performance targets and that Justice reported the number of cases in Indian Country. A program evaluation for this goal could compare the difference in gang activity on reservations where assistance and training is provided with reservations where assistance and training have not been provided. While program evaluations will also be hampered by the lack of underlying data about the drug- and gang-related violence, they might provide some indications of the comparative effectiveness of different interdiction programs.

Availability and/or Use of Illegal Drugs

Overall progress made by Justice toward achieving a reduction in the availability and/or use of illegal drugs is difficult to ascertain because it did not have fiscal year 2000 performance targets for two of five measures, and the relationship of one measure to the outcome was not clear. Justice did not have a performance target for measures related to drugs removed and its efforts to seize, dismantle, and dispose of clandestine laboratories. Justice indicated that the measure of the amount of drugs removed is to be discontinued because it does not adequately assess performance and is not results oriented. Justice explained that DEA could not estimate the amount of drugs to be removed by type because these vary from case to case. Rather, Justice reported that DEA seeks to investigate cases that will have the greatest impact on drug trafficking, drug-related crime, and violence and that drug seizures are merely a by-product of those investigations. Regarding not setting targets for its efforts to seize, dismantle, and dispose of clandestine laboratories, as previously noted, Justice does not want the public to perceive that it is pursuing arbitrary targets for the sake of meeting particular goals. Also, we noted that Justice’s fiscal year 2000 actual performance of 1,888 clandestine laboratories seized, dismantled, and disposed had decreased from its fiscal year 1999 actual performance of 2,024. According to Justice, it receives leads from state and local agencies or concerned citizens regarding the location of clandestine laboratories. The number of leads vary from year to year which results in a variance in the number of seizures in a given year. In addition, the performance report did not explain how positive responses for inquiries to Justice’s El Paso Intelligence Center (EPIC) contributes toward achieving the outcome to reduce the availability of drugs. As previously noted, we believe the usefulness of annual reports and plans could be improved by better articulating a results-orientation that would include explanatory information on goals and measures.

Justice fell short of achieving the performance targets for three measures. Justice fell somewhat short on its performance target to improve intelligence gathering. Specifically, Justice reported it had 22,624 inquiries to EPIC resulting in positive responses instead of the performance target of 24,602 inquiries. Justice reported that it did not meet its performance measure for the number of EPIC inquiries resulting in positive responses because EPIC did not receive as many requests for information as anticipated. Other unmet performance measures were to identify and dismantle major drug trafficking organizations. Specifically, Justice’s performance target was to identify 250 and dismantle 50 U.S.-based drug organizations. Justice reported that the FBI identified 201 major drug trafficking organizations and dismantled 12. Justice attributed this shortfall to the FBI overestimating what could be accomplished based on resource constraints. Justice also indicated that the Organized Crime Drug Enforcement Task Force (OCDETF) had a base reduction of agents and support staff at the end of fiscal year 2000 and that this would affect FBI’s ability to identify and dismantle major drug trafficking organizations in fiscal year 2001. Accordingly, the performance targets for fiscal year 2001 were revised.

In a July 1999 report, we stated that DEA did not have performance targets for disrupting and dismantling drug trafficking organizations. In the absence of such targets, it is difficult to assess DEA’s overall effectiveness in achieving its strategic goals. In our July 1999 report, we recommended that the Attorney General direct the DEA Administrator to work closely with Justice and the Office of National Drug Control Policy to develop measurable DEA performance targets for disrupting and dismantling drug trafficking organizations consistent with the performance targets in the National Drug Control Strategy. In response to our recommendation, DEA (1) developed a new strategic plan, which was approved in May 2000; (2) participated in a Justice work group to define the terms “disrupt” and “dismantle”; and (3) formed an internal GPRA Work Committee to assess and develop a feasible management approach to identify and establish quantifiable performance targets. Justice indicated in its fiscal year 2000 performance report that, under DEA’s new strategic plan, DEA developed another performance measure—percent of major drug trafficking organizations disrupted or dismantled—and is developing a process to capture information and data to report on this measure. Justice anticipates, however, that the system will take between 2 and 3 years to be fully operational.

Similar to the previous outcome, Justice’s performance report included a brief explanation about data collection and storage, data validation and verification, and any known data limitations. Except as noted above about DEA developing a new process for capturing data on the percent of major drug trafficking organizations disrupted or dismantled, none of the performance measures noted any data limitations.

Justice’s strategies and initiatives to reduce the availability and/or use of drugs generally seem reasonable and clear. Fiscal year 2001 performance targets were revised based on performance results in fiscal year 2000. Although Justice generally explained why it did not meet certain targets and revised those targets downward, its strategies do not articulate what Justice will do differently to achieve its unmet goals in the future. In addition, the strategies did not discuss determining the underlying reason for EPIC not receiving as many requests for information as anticipated—a piece of knowledge that might improve upon the relevancy, appropriateness, and usefulness of the performance measure and contribute to determining whether other measures might be more useful. Furthermore, Justice did not include specific strategies or goals for mitigating the implications of FBI and OCDETF resource constraints, including human capital management issues. Although Justice’s performance plan identifies agencies that have crosscutting activities related to reducing the availability and/or use of drugs, the plan does not discuss efforts in relation to achieving the outcome. For example, the plan has FBI performance measures and expects future DEA performance measures on dismantling drug trafficking organizations, but the plan does not adequately explain how joint planning and coordination will contribute to achieving the overall outcome to reduce illegal drug availability and/or use. According to Justice, interagency cooperation is key to successful drug enforcement, and Justice reported that it has developed a number of programs through which investigators can coordinate. However, the plan does not include strategies for enhancing or measuring the contribution of these programs to the overall achievement of the outcome. In our February 1999 report, we note that the listing of current programs and initiatives that were often included in agencies plans were useful for providing an understanding of what agencies do. However, presentations that more directly explain how programs and initiatives achieve goals would be most helpful to congressional and other decisionmakers in assessing the degree to which strategies are appropriate and reasonable.

Services Provided by the INS

Overall progress made by INS towards providing timely, consistent, fair, and high-quality services was difficult to fully gauge because the measures did not enable us to assess progress toward achieving this planned outcome. For example, the performance measures on the number of naturalization cases adjudicated, the percent of naturalization and benefit applications found on line, and the number of these applications filed on line do not indicate whether users of INS services are receiving timely, consistent, fair, and high-quality services. Regarding the performance target for achieving a 99-percent level of compliance with INS’ quality standards for naturalization applications, the performance report did not clearly explain what is covered in these standards. Therefore, it is unclear whether compliance with these standards is an indication of timely, consistent, fair, and high-quality services. Again, we believe that opportunities exist for Justice to improve the usefulness of its annual report and plan by better articulating a results-orientation that would include explanatory information on goals and measures.

Justice reported that it had not met its performance target for an average case processing time of 6 months for naturalization applications, instead reporting an average case processing time of 8 months during fiscal year 2000. Justice did not explicitly discuss the reason for missing the performance target for average case processing time, but implied that it was a resource issue. Specifically, Justice reported that during peak periods at the end of the year, INS met the targeted 6-month processing time for naturalization applications by shifting resources from other services to increase production. On the basis of its fiscal year 2000 performance, INS expects to achieve a case processing time of 9 months in fiscal year 2001 for naturalization applications. Furthermore, historical data in Justice’s performance report shows that improvement has been made to reduce the average case processing time for naturalization applications from 27 months in fiscal year 1998 to 8 in fiscal year 2000.

Similar to the previous outcomes, Justice’s performance report included a brief explanation about data collection and storage, data validation and verification, and any known data limitations. Justice’s performance report indicated data limitations and efforts to improve the accuracy and timeliness of the data. For example, Justice reported that in fiscal year 2001 INS’ naturalization case capability will be fully deployed under its Computer Linked Application Information System (CLAIMS 4) and will allow data for these cases to be fully automated and case-based, providing for timely and accurate data. In a May 2001 report, we said that aliens face long waits for a resolution to their case and have difficulty obtaining accurate information on how long they can expect to wait. We reported that INS did not know how long it took to process aliens’ applications because the agency’s automated application data were incomplete and unreliable. Specifically, we reported that INS’ available servicewide automated systems contained unreliable data and its districts did not have automated systems for tracking many types of applications. We pointed out that, in the absence of information on actual processing times, INS had been estimating processing times, but that the usefulness of the estimation method was limited. We recommended, and Justice agreed, that INS develop the capability to begin to calculate and report actual processing times for applications as soon as reliable automated data are available from its servicewide systems, CLAIMS 3 and CLAIMS 4. Justice’s performance report states that on the basis of its fiscal year 2000 performance, it expects to meet the corresponding 2001 targets for average case processing times of 9 months for naturalization applications and 14 months for adjustment of status applications and expects to meet the 2001 performance target of 99 percent compliance with quality standards for naturalization applications.

Justice strategies and initiatives do not sufficiently discuss achieving the outcome to provide timely, consistent, fair, and high-quality services. The strategies primarily address maintaining or improving application process times and generally do not discuss consistent, fair, and high-quality services. As previously noted, INS has quality standards that it is using as a measure, but the performance plan does not articulate the specific quality standards for achieving the outcome. Justice did not discuss the implications of using strategic human capital management as a strategy to help achieve this outcome even though one performance target was not achieved until resources were realigned. Thus, the deployment of available staff appears to be critical to achieving the timeliness performance targets. In addition, the performance plan did not provide as much detail as it could have to describe INS’ strategy to provide electronic filing of applications and the implications for accomplishing timely, consistent, fair, and high-quality services. Although Justice identified that it is deploying CLAIMS 4 software to field offices in fiscal year 2001 and upgrading CLAIMS 3 automated support, the performance report does not explain how the information technology improvements will contribute to achieving better INS services. In our February 1999 report, we note that the listing of current programs and initiatives that were often included in agencies’ plans were useful for providing an understanding of what agencies do. However, presentations that more directly explain how programs and initiatives achieve goals would be most helpful to congressional and other decisionmakers in assessing the degree to which strategies are appropriate and reasonable.

Securing U.S. Borders From Illegal Immigration

Overall progress made by INS towards achieving this outcome is difficult to fully gauge because INS has a new performance measure for which there was no fiscal year 2000 performance target, and the other two performance measures did not enable us to assess progress toward achieving this planned outcome. Because it was a new measure, Justice did not set a performance target for high-priority border corridors demonstrating optimum deterrence, a critical performance measure to determine whether it is securing U.S. borders. Justice did not discuss the rationale for the new measure or how the new measure will better enable INS to assess its progress toward securing our borders. Justice said that during fiscal year 2000, INS continued to refine the border control operational effectiveness measure, in particular by using “corridors” rather than zones, with each sector identifying the corridors within their area of operation. Even though this was a new measure, Justice provided historical data from fiscal years 1994 to 1999, provided actual performance for fiscal year 2000, and projected performance targets for fiscal years 2001 to 2004 for this performance measure as an indication of its progress. The historical data showed that INS has maintained optimum deterrence in 6 of 26 corridors along the Southwest border during fiscal years 1998, 1999, and 2000.

In addition, while Justice met the targets for the other two performance measures for this outcome, these measures were not directly linked to the outcome because they omitted some aspects of the performance. For example, to deter illegal immigration at the source, INS has a performance measure to intercept undocumented offshore travelers en route to the United States. Justice reported that these intercepts were accomplished as a result of INS officers working closely with their host country government agencies, diplomatic missions, and others to provide advice, training, and assistance. Justice’s performance report does not sufficiently discuss the working relationship with host countries and others or the quality of INS officers’ training and assistance in relation to deterring illegal immigration at the source. In a May 2000 report, we said that the agency does not believe that overseas efforts have produced long-term impacts because training that INS personnel provided to foreign air carrier and law enforcement personnel diminished within a few months. In addition, INS believes that the non-INS personnel they have trained do not receive continual encouragement and support to perform their jobs professionally and, therefore, revert to their old practices.

In its performance report Justice stated that it expects to meet the 2001 performance targets for the three performance measures. Specifically, Justice’s performance report states that on the basis of fiscal year 2000 actual performance for demonstrating optimum deterrence in six corridors, INS expects to meet the 2001 performance target for this measure of deterrence in eight corridors. Justice exceeded, by a significant number, its performance targets for intercepting undocumented offshore travelers en route to the United States (the performance target was 8,283 interceptions and the actual was 19,007) and the offshore prosecutions assisted by INS (the performance target was 107 prosecutions and the actual was 514). Justice reported that improved and thorough reporting of these activities by the INS overseas district offices resulted in a more accurate account of these activities than had been available in the past, and INS significantly exceeded its targets. Justice’s performance plan indicates INS expects to also exceed the fiscal year 2001 performance targets of 9,324 interceptions and 119 prosecutions.

Although Justice briefly explained its data collection and storage, and data validation for each performance measure, there is some question about the credibility of the performance data. Specifically, Justice reported that all three performance measures related to securing the U.S. borders from illegal immigration had data limitations. With respect to measuring border corridors demonstrating optimum deterrence, the report states that collecting data to measure this goal is currently an intensive manual process and that INS is implementing a process to standardize recording and reporting of data to ensure consistency and validity. Justice also reported that data provided for corridors demonstrating optimum deterrence prior to fiscal year 1999 were estimated because data was not available for corridors during that period and that projected corridor effectiveness for fiscal years 2001 through 2004 is dependent on sufficient allocation of resources.

Justice’s strategy to secure U.S. borders from illegal immigration does not adequately discuss integration of resources to achieve the outcome. Justice’s basic strategy is to apply increased levels of Border Patrol staff, technology, and other resources in the busiest areas until the risk of apprehension is high enough to be an effective deterrent, thus creating acceptable areawide control. Justice’s performance plan does not discuss the mix of staffing, equipment, and technology needed to achieve the desired level of deterrence in each area nor does it clearly discuss the basis upon which a determination is made that a particular corridor has achieved optimum deterrence. In addition, in December 1999, we reported that INS had had difficulties attracting and retaining qualified applicants for Border Patrol positions. Justice’s performance report indicates that INS did not meet its performance target to have 9,377 Border Patrol agents on board at the end of fiscal year 2000, falling short by 196 agents. According to Justice, additional Border Patrol resources (personnel and technology) are needed in fiscal year 2002 to maintain and extend control along the border. INS expects to meet its hiring goals for agents in fiscal year 2001, reaching an on-board strength of 9,807 and projected an on- board strength of 10,377 agents for fiscal year 2002. According to Justice, INS has overcome difficulties in hiring Border Patrol agents. The Justice report also states that INS set records in fiscal year 2000 for the number of qualified applicants and the number passing the required tests. However, Justice did not discuss actions planned to bringing agents on board nor did it explain how having more applicants and candidates passing tests will ultimately result in achieving the targeted level of agents to be on-board in fiscal year 2001. While hiring more agents is a first step, INS did not discuss training and deployment of hired agents--initiatives that also impact on INS’ ability to have agents at the border. Furthermore, the performance plan did not explain Justice’s plans for obtaining equipment and technology needed to implement the border control strategy. As previously mentioned, strategies can be more useful if they describe how they will enable the agency to achieve its goals.

Justice stated in its 2002 performance plan that it continuously evaluates the effectiveness of its border control strategies, particularly for the Southwest border, and quarterly evaluates progress through the Commissioner’s Performance Management Reviews. In addition, Justice reported that several special studies have been initiated and are ongoing to evaluate border enforcement effectiveness. In a December 1997 report on the Southwest Border Strategy, we recommended a comprehensive and systematic evaluation plan of INS’ border strategy be developed to obtain information about the effectiveness of the strategy in reducing and deterring illegal entry. Justice anticipates conducting additional studies related to the effectiveness of INS’ enforcement activities at the border that includes one that responds to our recommendation. In a May 1999 report, we concluded that information on INS studies was too limited for us to assess whether these studies will provide the information needed to comprehensively and systematically evaluate the effectiveness of the strategy.

Comparison of Justice’s Fiscal Year 2000 Performance Report and Fiscal Year 2002 Performance Plan With the Prior Year Report and Plan for Selected Key Outcomes

For the selected key outcomes, this section describes strengths or remaining weaknesses in Justice’s (1) fiscal year 2000 performance report in comparison with its fiscal year 1999 report and (2) fiscal year 2002 performance plan in comparison with its fiscal year 2001 plan. This section also discusses the degree to which the agency’s fiscal year 2000 report and fiscal year 2002 plan addresses concerns and recommendations by us and Justice’s OIG.

Comparison of Performance Reports for Fiscal Years 1999 and 2000

We identified several strengths in Justice’s performance reports. First, both fiscal years 1999 and 2000 performance reports generally included (1) a comparison of actual performance with the projected level of performance (when a goal had a performance projection) as set out in the performance goals and (2) an explanation for why the goal was not met, where a performance goal was not achieved. Second, a key improvement of the fiscal year 2000 report was that, as required under GPRA, the report showed that Justice generally reassessed fiscal year 2001 performance targets on the basis of its performance for fiscal year 2000. Third, Justice’s performance report included historical data to provide perspective on its progress. And finally, we also noted that this year Justice issued a combined fiscal year 2000 performance report and fiscal year 2002 performance plan. The information is now presented in a sequential manner, discussing the results of the past year, then the anticipated performance for the current year, and finally the impact of next year’s performance. Presenting the information in this manner, we believe, provides decisionmakers with a better understanding of the agency’s progression toward achieving its goals.

The fiscal year 2000 performance report, like the 1999 report, also contains several weaknesses in that it does not consistently address changes in the performance report as to why certain measures were discontinued, new ones added, or revisions made to existing measures. For example, INS changed its performance goal to measure deterrence in relation to corridors rather than zones; however, there is no discussion as to the rationale for changing the areas of operation to be measured or how the new measure will better enable INS to assess its progress toward securing our borders. We believe an explanation in the performance report would be useful to better understand the relationship of revised or new goals and measures toward achieving the performance goal. Furthermore, although Justice generally explained why it did not meet certain targets and revised those targets downward, its strategies do not articulate what Justice will do differently to achieve its unmet goals in the future.

In addition, eight management challenges identified in our 2000 GPRA report continue to be management challenges. Progress in resolving two of the eight management challenges—INS restructuring and internal control weakness at DEA—was not discussed in either the 1999 or 2000 performance report.

Comparison of Performance Plans for Fiscal Years 2001 and 2002

Our comparison of Justice’s performance plans for fiscal years 2001 and 2002 found that many of the weaknesses previously identified by us in the 2001 plans were present in the 2002 plan. Although the strategic human capital management goals and measures were identified in its 2001 and 2002 performance plans, Justice’s 2002 performance plan did not address human capital strategies in relation to achieving programmatic outcomes for two goals—dismantling major drug trafficking organizations and providing timely processing of naturalization applications. In addition, Justice identified personnel skills that supported each strategic goal, but did not discuss whether it had the staff with these skills or whether it needed to acquire or develop staff to meet agency needs. The performance plans also consistently identified program evaluations related to each of its strategic goals. However, the plans do not discuss whether these evaluations could be used in assessing the achievement of goals or as alternative measures for performance. For example, we noted that Justice reported on a program evaluation to verify and validate CLAIMS, which supports INS’ benefit processing that may provide information to improve operations, but will not provide outcome measures. However, we noted that another program evaluation concerning a multiyear study of the employment verification pilots may provide some information toward measuring progress. We believe that providing information on how the program evaluations would help measure Justice’s achievement of its outcomes could be useful to decision-makers.

Like its fiscal year 2001 performance plan, Justice’s fiscal year 2002 performance plan consistently identified crosscutting activities, but its discussion of crosscutting activities generally did not discuss how the activities could be coordinated to improve overall performance within Justice. For example, to disrupt and dismantle major drug trafficking criminal enterprises, the performance plan states that interagency cooperation is key to successful drug enforcement and provides information on a number of programs through which investigators from various agencies can coordinate. The plan cites DEA, FBI, the Criminal Division, and other federal law enforcement agencies as participants in these programs. However, the plan does not explain how the strategies of Justice’s components are mutually reinforcing, nor does it explain common or complementary performance indicators.

Justice stated that it has developed new performance measures for goals where in the past it had not set targets. Most of Justice’s performance measures in its 2002 performance plan had targets against which to measure progress. However, we do not know the extent to which the new performance measures will clearly demonstrate results achieved. As mentioned earlier, in our September 2000 report, we provided information and examples to assist agencies in identifying how they might use evaluations to improve their performance reporting. We noted that program evaluations are objective, systematic studies that answer questions about program performance and results. An evaluation study can explore the benefits of a program as well as ways to improve program performance by examining a broader range of information than is feasible to monitor on an ongoing basis through performance measures. For example, a program evaluation was conducted of an INS border control initiative in El Paso, Texas, called Operation Hold the Line. Operation Hold the Line was a new INS enforcement approach introduced in 1993. Rather than apprehending aliens after they had illegally crossed the border in El Paso, INS sought to prevent illegal entry from occurring in the first place by increasing the number of Border Patrol agents in El Paso and position them in high visible locations along the border. The evaluation collected data to assess the effects of Operation Hold the Line on a number of outcomes, including illegal and legal crossings, business activity, crime, education, births, and the use of social services in El Paso. By collecting quantitative and qualitative data on a range of outcome indicators, the researchers were able to draw conclusions about the representativeness, scope, and magnitude of the Operation’s effects.

The fiscal year 2002 performance plan clearly identified the OIG management challenges and Justice designated areas of material weaknesses. The plan does not consistently identify our recommendations or concerns in relation to achieving goals and performance measures. For example, the Justice plan provided information on actions taken to address our recommendation to DEA regarding its performance measures but the performance plan did not address actions taken in response to our recommendation to INS regarding its estimation for application processing times.

Justice’s Efforts to Address Its Major Management Challenges Identified by GAO

We identified two governmentwide high-risk areas: strategic human capital management and information security. Regarding strategic human capital management, Justice’s performance plan had goals and measures related to human capital, and the agency’s performance report explained its progress in resolving human capital challenges. With respect to information security, Justice’s performance plan had goals and measures related to information security, and the agency’s performance report explained its progress in resolving its information security challenges.

In addition, we identified 12 major management challenges facing Justice. Justice’s performance report discussed the agency’s progress in resolving many of its challenges, but it did not discuss the agency’s progress in resolving the following challenges: (1) internal control weaknesses at DEA, (2) options for restructuring INS, (3) weaknesses in Justice’s asset forfeiture program, and (4) program management weaknesses in the Weed and Seed program. As shown in table 1, of the agency’s 12 major management challenges, its performance plan (1) had goals and measures that were directly related to five challenges; (2) had a goal but no measures that were directly related to one challenge; (3) had goals and measures that were indirectly applicable to one of the challenges; (4) had no goals and measures related to two of the challenges, but discussed strategies to address them; or (5) had no goals, measures, or strategies to address three of the challenges. Appendix I provides detailed information on how Justice addressed these challenges and high-risk areas as identified by us and its OIG.

Scope and Methodology

As agreed, our evaluation was generally based on the requirements of GPRA, the Reports Consolidation Act of 2000, guidance to agencies from the Office of Management and Budget (OMB) for developing performance plans and reports (OMB Circular A-11, Part 2), previous reports and evaluations by us and others, our knowledge of Justice’s operations and programs, GAO identification of best practices concerning performance planning and reporting, and our observations on Justice’s other GPRA- related efforts. We also discussed our review with officials in the Department of Justice and its OIG. The agency outcomes that were used as the basis for our review were identified by the Ranking Minority Member of the Senate Committee on Governmental Affairs as important mission areas for the agency and do not reflect the outcomes for all of Justice’s programs or activities. The major management challenges confronting Justice, including the governmentwide high-risk areas of strategic human capital management and information security, were identified in our January 2001 performance and accountability series and high-risk update and were identified by Justice’s OIG in December 2000. We did not independently verify the information contained in the performance report and plan, although we did draw from our other work in assessing the validity, reliability, and timeliness of Justice’s performance data. We conducted our review from April through June 2001 in accordance with generally accepted government auditing standards.

Agency Comments

We discussed our draft report with Justice officials on June 13 and 14, 2001, and received written comments on June 19, 2001. The full text of Justice’s written comments is included in appendix II. In its letter, Justice discussed four major areas—our report’s overall focus, outcome goals that were evaluated, limited consideration of new goals and measures in its performance plan, and the presentation of progress on management challenges.

Overall focus. Justice believes that our report focuses heavily on what its performance report and plan do not discuss, on targets not established, or on targets not met. In addition, Justice believes that our report does not focus on improvements the agency has made. For example, in addressing INS’ goal for ensuring that immigration benefit services are timely, fair, and consistent, Justice said that our report focused on the agency missing the naturalization case processing time by 2 months. Justice believes that reaching an 8-month processing time is an incredible achievement, given that the processing time was 27 months just 2 years earlier.

We addressed many of the improvements that Justice made under the section comparing the performance report and plan with the previous year’s report and plan. The sections discussing achieving outcomes are an assessment of progress toward achieving results on the basis of performance measures and targets, historical data, and our work related to program areas. We acknowledge that Justice’s performance report included historical data for fiscal years 1998 and 1999 for many of its measures. But other than the example Justice cited in its comments, the historical data for the other measures did not clearly demonstrate improvements. Furthermore, after meeting with Justice to discuss the draft report, we revised the text to include the historical data regarding reported improvement to reduce the processing time of naturalization cases (the example Justice cited).

Outcome goals. Justice acknowledged that the outcomes we used in our analysis were prescribed by the requestor, but believes that the outcome of reduced availability and/or use of illegal drugs is not part of its mission. Instead, Justice stated that its strategic goal relative to enforcing the nation’s antidrug laws is to reduce the threat and trafficking of illegal drugs by identifying, disrupting, and dismantling drug trafficking organizations. Thus, Justice does not believe that any of its annual goals or measures will relate directly to the achievement of the outcome to reduce the availability and/or use of illegal drugs, giving the false impression that its report and plan are deficient. Justice also believes that, while reducing drug- and gang-related violence is part of its mission, only four of its measures under this outcome, not nine, should have been used in our analysis. Thus, Justice believes that the other five measures in assessing this outcome should be deleted.

In our opinion, reducing the threat and trafficking of illegal drugs by identifying, disrupting, and dismantling drug trafficking organizations is directly related to reducing the availability and/or use of drugs. If drug trafficking organizations are disrupted and dismantled, clearly this will affect the availability of drugs on the street. Concerning Justice’s contention that five of the measures under the planned outcome to reduce drug- and gang-related violence should be deleted, we disagree. The five measures in question are (1) number of criminal background checks, (2) number of persons with criminal backgrounds prevented from purchasing firearms, (3) number of cases in Indian Country, (4) number of new Interpol cases, and (5) number of new treaties with other countries. In its performance report, Justice included these five measures under its strategic goal to reduce the threat, incidence, and prevalence of violent crime, especially as it stems from gun crime, organized crime, and drug and gang-related violence. While Justice’s strategic goal is not an exact match to the planned outcome, we believe that the performance measures included in our analysis are appropriate. Specifically, all of these measures, in our opinion, have the potential, in part, to be related directly or indirectly to drug- or gang-related violence. For example, members of gangs may be prevented from purchasing firearms because of criminal background checks. Additionally, under the measure for the number of cases in Indian Country, growing juvenile gangs is one of the major issues discussed in Justice’s performance report. Likewise, Justice’s performance report indicated that international law enforcement cooperation is critical to addressing the dramatic growth of transnational crime such as narcotics trafficking and terrorism, which may, in part, relate to drug- and gang-related violence on an international level.

New goals and measures. Justice believes that its performance measures have matured over time and indicated that it has discontinued old and added new measures as appropriate. Since some measures are new in fiscal year 2002, performance targets did not exist for the fiscal year 2000 plan. Nevertheless, Justice reported on its accomplishments by providing historical data for fiscal years 1999 and 2000 whenever possible. Justice believes that this is particularly significant for two outcomes: (1) timely, consistent, fair, and high-quality services provided by INS and (2) U.S. borders secure from illegal immigration. Concerning the first outcome, Justice noted that it has a new performance measure for the level of compliance with quality standards for processing naturalization cases, and Justice reports that it has achieved 99 percent compliance with those standards since fiscal year 1999. Concerning the second outcome, Justice believes that it is unfair for us to report that its performance was not considered sufficient to assess progress merely because there was no performance target against which to measure. Furthermore, Justice believes that it is particularly discouraging for the managers and analysts that worked to develop a measure which Justice believes is an excellent example of reporting outcomes.

While we used performance measures and targets in our analysis, our evaluation was also based on other factors, such as previous reports and evaluations by us and others, our knowledge of Justice’s operations and programs, and our identification of best practices concerning performance planning and reporting. Furthermore, we added text to reflect the development of new measures in our comparison of the performance report and plan with the previous year’s report and plan. Concerning the quality standards issue, we did not indicate that INS had not met the 99- percent standard. We merely indicated that the performance report does not explain what is covered by these standards. While Justice believes that the existence of quality standards implicitly explains the relationship to the outcome, we believe that the report could articulate what aspects of quality service have been achieved. Moreover, we included information from our previous work relative to CLAIMS that indicated that data limitations could affect Justice’s assessment of the quality of services.

Management challenges. Justice believes that the format used in appendix I of our report did not lend itself to an accurate description of its performance report and plan. In the column describing how Justice assessed its progress in resolving the management challenges, we sometimes indicated that progress relative to a management challenge was not discussed. Justice believes that, although technically correct, this is misleading because information about some of the management challenges may be included in the third column of the table, which discusses applicable goals and measures in Justice’s performance plan. Justice suggested that, where appropriate, we should indicate in the report column that information about a particular challenge is included under the “plan” column. Further, Justice noted that, in some instances, more complete discussions of its progress in addressing some issues can be found in other, more applicable, documents.

We included text in the first paragraph of appendix I to explain that Justice did not have performance goals and measures for fiscal year 2000 to assess progress and that, for some of the management challenges, Justice discussed the challenge in its fiscal year 2002 performance plan. We did not verify whether additional information about Justice’s management challenges may be found in other documents, but if information in other documents is relevant to assessing Justice’s progress, it would have been useful to decisionmakers if Justice were to identify these documents in the performance report.

In addition to the four areas discussed above, Justice raised one final issue. Justice noted that it continues to face conflicting pressures to keep its performance report and plan streamlined and yet to include more detailed information. Justice characterized our position as one desiring considerably more detail in its performance report and plan. Our point is not necessarily that Justice needs to include more detailed information, but rather that it needs to better articulate and explain how performance and strategies relate to achieving desired goals.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this report. At that time, we will send copies of this report to appropriate congressional committees; the Attorney General; and the Director, Office of Management and Budget. Copies of this report will also be available to others on request.

If you or your staff have any questions, please call me at (202) 512-8777. Key contributors to this report were Linda Watson, Tim Outlaw, Mary Hall, Julia Duquette, David Irvin, and Charles Vrabel.

Appendix I: Observations on the Department of Justice’s Efforts to Address Its Major Management Challenges

The following table identifies the major management challenges confronting the Department of Justice (Justice), which includes the governmentwide high-risk areas of strategic human capital management and information security. The first column lists the 19 management challenges identified by our office and/or Justice’s Office of the Inspector General (OIG). The second column discusses what progress, as discussed in its fiscal year 2000 performance report, Justice made in resolving its challenges. The third column discusses the extent to which Justice’s fiscal year 2002 performance plan includes performance goals and measures to address the challenges that we and the OIG identified. We found that Justice’s performance report discussed the agency’s progress in resolving many of its challenges, but it did not discuss the agency’s progress in resolving the following challenges: (1) internal control weaknesses at the Drug Enforcement Administration (DEA), (2) options for restructuring the Immigration and Naturalization Service (INS), (3) weaknesses in Justice’s asset forfeiture program, (4) program management weaknesses in the Weed and Seed program, (5) proper management of grant funds, and (6) enforcement efforts along the northern border. However, Justice officials pointed out that progress for these management challenges are not discussed in the fiscal year 2000 performance report because there were no goals, measures, or strategies in its fiscal year 2000 performance plan. Furthermore, Justice noted that some of these management challenges are included in its fiscal year 2002 performance plan.

Of the agency’s 19 major management challenges identified by us and Justice’s OIG, Justice’s performance plan (1) had goals and measures that were directly related to ten challenges; (2) had a goal but no measures that were directly related to one challenge; (3) had goals and measures that were indirectly applicable to two of the challenges; (4) had no goals and measures related to two of the challenges, but discussed strategies to address them; or (5) had no goals, measures, or strategies to address four of the challenges.

Appendix II: Comments from the U.S. Department of Justice