Grants Management: Despite Efforts to Improve Weed and Seed Program Management, Challenges Remain
Released: 2004-03-24T00:00:00

Letter

Background

The Weed and Seed program is a Justice discretionary grant program within OJP. The Assistant Attorney General for OJP is responsible for the overall management and oversight of offices within OJP, including EOWS. The Assistant Attorney General for OJP sets policies, promotes coordination among OJP bureaus and offices, and ensures that EOWS follows its policies and procedures. EOWS provides funding to grantees to help prevent and control crime and improve the quality of life in targeted high-crime neighborhoods across the country. It is a joint federal, state, and local program for coordinated law enforcement and neighborhood reinvestment. Federal program funding is to support Weed and Seed sites and to provide training and technical assistance.

The Weed and Seed program has grown since it began in fiscal year 1991 with three pilot sites in Kansas City, Missouri; Omaha, Nebraska; and Trenton, New Jersey, and a relatively small investment of federal money. From fiscal year 1999 through 2003, the number of Weed and Seed sites increased from 163 to 221, while the total annual program budget generally increased from about $45 million to $70 million. See table 1 for fiscal years 1999-2003 data on the Weed and Seed program, including EOWS’s funding history. In fiscal year 2003, with a budget of about $70 million, EOWS awarded grants to 221 Weed and Seed sites.

EOWS is responsible for the national management and administration of the Weed and Seed program, including developing policy and providing guidance and oversight. EOWS currently administers the Weed and Seed program with a staff of 2 management officials, 10 grant monitors, 3 support staff, 3 detailees, and 6 contractors.

Before a community can apply for Weed and Seed grant funding and become an eligible site, it must first apply for and gain official recognition from EOWS. In order to obtain official recognition, a potential Weed and Seed site must show in its application that it has a strategy for weeding and preventing crime. Once the application has been received, EOWS creates and maintains the official recognition files that are intended to include documentation such as the rationale for decisions to grant or deny official recognition to an applicant. Official recognition requires the U.S. Attorney in the area where the Weed and Seed site is to be located to organize a local steering committee made up of various federal, state, and local representatives, including residents, to be responsible for local administration of the program. For official recognition, a site is also required to develop a management plan, engage residents and other partners in its activities, and develop a comprehensive program to weed out crime and gang activity and seed the area with social services, economic services, and economic revitalization.

Weed and Seed program guidance requires that its sites show plans for addressing four required elements: (1) law enforcement; (2) community policing; (3) crime prevention and intervention, and substance abuse prevention, intervention, and treatment; and (4) neighborhood restoration. According to EOWS, law enforcement should attempt to eliminate the most violent offenders by coordinating and integrating the efforts of federal, state, and local law enforcement agencies in targeted high-crime neighborhoods. The objective of community policing is to raise the level of citizen and community involvement in crime prevention and intervention activities. Crime and substance abuse prevention, intervention, and treatment should include youth services, school programs, community and social programs, and support groups. Finally, neighborhood restoration should focus on assistance to distressed neighborhoods through economic and housing development. Weed and Seed sites fund a variety of law enforcement and community activities. For example, law enforcement- funded activities range from participation in a multijurisdictional task force to conducting bike patrols in the community. See appendix II for additional information on Weed and Seed activities at the sites GAO visited.

Applicants that are officially recognized are eligible to apply for funding for up to 5 years. OJP’s Office of the Comptroller creates and maintains official grant files for sites that are awarded funding by EOWS. Applicants are rated based on the strategy they developed in response to the four required elements, as stated above.

In our 1999 report, we made four recommendations to help EOWS improve program monitoring and management. The first was to develop adequate internal controls to fully document significant qualification and funding decisions. We found that EOWS lacked internal controls requiring significant program management decisions be documented. Therefore, EOWS was not able to ensure that it was making the best decisions about allocating available funds. Our second recommendation was to improve program monitoring to ensure that sites met the grant requirement of submitting progress reports and that EOWS site visits were documented. We found that EOWS did not always ensure that local sites submitted progress reports and that grant monitors documented the results of site visits. Such documentation would help EOWS management and grant monitors determine how sites are meeting program objectives and how well sites are complying with grant requirements, and assist them in making future grant qualification decisions. Our third recommendation was to develop criteria for determining when sites have become self- sustaining and when to reduce or withdraw program funding. We found that although self-sustainability is central to the program, no site’s funding had been reduced or withdrawn as a result of its efforts to become self- sustaining in the 9 years of the program’s existence. Establishing such criteria is important because the Weed Seed program was founded on the premise that federal funding would continue for a finite period, after which a Weed and Seed site would be self-sustaining. Our fourth recommendation was to develop additional performance measures to track program outcomes. We found that EOWS’s performance indicators generally tracked activities rather than program results; therefore, EOWS was not able to measure the success of the program.

EOWS generally agreed with three of the four recommendations presented in the report and discussed future actions it planned to take. EOWS officials disagreed with our recommendation on self-sustainability. They stated that developing criteria to ascertain self-sustainment is redundant since EOWS adopted a 5-year rule under which it could discontinue awarding funding to qualifying sites unless the sites expanded to an additional neighborhood site.

Internal Controls Have Been Developed, but Challenges Remain

Although EOWS has developed internal controls intended to require that significant qualification and funding decisions be documented and readily available for review as we recommended in 1999, these policies and procedures are generally not being followed in the files we reviewed. In response to our 1999 recommendation on internal controls, EOWS established policies and procedures intended to ensure that significant qualification decisions were documented. In addition, OJP requires that qualification and funding decisions are to be documented in official recognition and official grant files. Our review of 20 official recognition and 10 official grant files showed that some of EOWS’s official recognition and official grant files were missing full documentation regarding the qualification and funding decisions. However, before our file review ended, EOWS officials produced further documentation, which they acknowledged was not in the official recognition and official grant files as required, but rather in the personal working files of grant monitors and thus not readily available to EOWS management and external reviewers. This lack of ready availability is not in keeping with EOWS’s and OJP’s policies and procedures, or with the Comptroller General’s standards for internal controls. Without having official recognition and official grant files complete and readily available, it may delay and complicate EOWS officials’ oversight of the documentation of qualification and funding decisions.

Policies and Procedures Have Been Developed to Document Significant Decisions

Both EOWS and OJP policies and procedures have been developed to help ensure the documentation of significant decisions. In 2000, EOWS developed a policies and procedures guide in response to the recommendation we made in our 1999 report intended to ensure that significant qualification and funding decisions for new and existing sites were always fully documented, and further revised the guide in 2003. This guide requires that all documentation pertaining to official recognition decisions be kept in the official recognition files. EOWS is also required to follow the policies in the OJP Grants Management Policies and Procedures Manual, which requires that EOWS fully document program management decisions in both official recognition files and official grant files. The OJP manual specifically requires that the official recognition files and official grant files contain key documents such as ratings information, rejection letters, and applications.

Official Recognition and Official Grant Files Were Incomplete

Despite EOWS’s and OJP’s efforts to require the documentation of qualification and funding decisions in official recognition and official grant files, some of EOWS’s official recognition and official grant files we reviewed were incomplete at the time of our file review. In reviewing the official recognition files, we randomly selected 10 files to review from the 94 files submitted for official recognition in fiscal year 2002. Additional information about how we selected the files is in appendix I. We found sufficient documentation in 7 of the 10 files to determine the basis and rationale for decisions to award official recognition. However, in the remaining 3 files we were unable to determine the basis and rationale for such decisions because documentation was insufficient. For example, key documents such as the ratings information that EOWS grant monitors use to record their assessment of the official recognition applications did not contain the basis and rationale for the award decision. Additionally, OJP’s grant policies and procedures require EOWS’s grant monitors to prepare a rejection letter, informing applicants of reasons for rejection when funding is denied and place a copy of the rejection letter in the official recognition file. We reviewed all 10 official recognition files for the applicants that were eligible to apply for funding in fiscal year 2002 but were rejected. We found a rejection letter was missing in 7 of the 10 files.

We also reviewed documents in EOWS’s official grant files to assess EOWS’s efforts to comply with EOWS and OJP policies for fully documenting funding decisions. We randomly selected 10 official grant files to review from the 31 sites that were funded in fiscal year 2002. We found that none of the 10 official grant files fully documented funding decisions for Weed and Seed sites. For example, the application, a basic component of the official grant file, was missing in 8 of the 10 files.

Challenges Persist in Making Documentation Readily Available

Several directives require that grant documentation be readily available. OJP’s policies and procedures manual requires that documentation be kept readily available in the official recognition and official grant files, so that OJP and EOWS management can identify and resolve any problems or deficiencies in grantees’ compliance with relevant policies and procedures. OJP’s policies and procedures manual requires that all pertinent information that should be in the official recognition and the official grant files be kept in a centralized location to facilitate reviewing for completeness. In addition, the Comptroller General’s guidance on internal controls in the federal government, Standards for Internal Control in the Federal Government, requires that the documentation for all transactions and significant events be readily available for examination. The Assistant Attorney General for OJP is responsible for holding EOWS accountable for following its policies and procedures. In reference to the ratings information and rejection letters missing from the official recognition files, EOWS produced the documents after our initial review, informing us that the documents were kept in the personal files of the grant monitors--rather than the official recognition files--a practice not in keeping with EOWS’s and OJP’s policies and procedures. In reference to the missing applications, EOWS officials also produced them after our initial review, but did not specify where they found them. Not having complete and readily available official recognition and official grant files may delay and make it more difficult for OJP and EOWS officials to perform management functions such as overseeing and assessing the documentation and oversight of qualification and funding decisions. While we eventually were able to obtain documentation showing the basis and rationale for EOWS decision making, the incomplete official recognition and official grant files made the review difficult and time-consuming.

In responding to a draft of this report, Justice stated that electronic documents maintained in its grant management system (GMS) are considered a part of the official recognition and official grant files and are readily available to EOWS, and that they should have been readily available to, and easily accessible by, GAO.

EOWS Has Taken Steps to Improve Program-Monitoring Documentation, but Challenges Remain

In response to our 1999 recommendation, EOWS has taken steps to improve program monitoring, as with the documentation of qualification and funding decisions, but challenges remain in making the monitoring documentation readily available. While OJP requires progress reports and site visit reports to be included in the official grant files, our review in 2003 of 10 official grant files showed that some of EOWS’s files were missing full documentation. However, before our review ended, EOWS provided further documentation that fully documented progress reports and site visits, which it acknowledges was not in the official grant files, and thus not readily available. This lack of readily available documentation is not in keeping with OJP’s policies and procedures. Not having complete and readily available official grant files may delay and complicate EOWS officials’ and external reviewers’ assessment of whether EOWS’s monitoring requirements are being followed.

Official Grant Files Were Incomplete

Despite EOWS’s efforts, some of the official grant files we reviewed were incomplete at the time of our file review. We randomly selected 10 files from the 31 sites that were first funded in fiscal year 2002 to determine whether the sites had submitted the required progress reports from January 1, 2003, through June 30, 2003. Of the 10 files we reviewed, 6 did not contain any of the required progress reports for the period. The remaining 4 files included the required progress reports. OJP’s policies and procedures require EOWS to ensure that progress reports are included in the official grant files. However, when we asked EOWS about the missing progress reports, EOWS provided us with the requested documents. We asked, but EOWS did not specify where it found the missing documentation.

In addition, EOWS’s grant monitors are required to conduct site visits a minimum of every 18 months and document their visits. This documentation is meant to convey to EOWS management officials how well sites are complying with grant requirements and is to be used by EOWS in making funding decisions. Grant monitors are to prepare a report of the visits and forward a copy to the Office of the Comptroller for inclusion in the official grant file. We reviewed the same 10 grant files discussed above to determine whether site visits were fully documented. We found that a site visit report was not yet due for 7 of the 10 files. In the remaining 3 files where a site visit was due, none of the 3 files contained documentation that the site visit had been conducted. We requested to speak to the responsible grant monitors to determine whether site visits had been conducted. EOWS officials told us that the monitors were no longer involved with the program but provided additional documentation for the 3 files we reviewed. The documentation showed that of the 3 sites, only 1 visit had been conducted; the other 2 were scheduled but had been delayed. While we only reviewed 3 files where a site visit was due, the lack of documentation in all 3 files, as well as the fact that only 1 of 3 site visits had been conducted, added to our concerns about the completeness of official grant files for program oversight.

Challenges Persist in Making Monitoring Documentation Readily Available

As we have previously mentioned, OJP’s policies and procedures manual and the Comptroller General’s guidance on internal controls require that documentation should be kept readily available for examination. Subsequent to our finding missing progress reports and site visit reports, EOWS provided us with additional documentation that demonstrated that the progress reports and one of the site visit reports had been completed. When asked about where they found the missing progress reports and site visit reports, EOWS officials did not specify where they found the missing documentation.

Despite Some Steps, Challenges Remain in Developing Criteria for Self-Sustainability

EOWS has established a rule and set some activities for sites to complete to encourage them to become self-sustaining. However, it has still not fully developed criteria to determine when sites have become self-sustaining and when to reduce or withdraw Weed and Seed grant funds, as we recommended in 1999. To become self-sustaining, sites must leverage additional resources from sources other than EOWS to sustain the Weed and Seed program without EOWS funds. We found that EOWS has not reduced or withdrawn funds from any Weed and Seed sites for reasons related to becoming self-sustaining. This is important because without overall program funding increases, new sites cannot be funded unless funding can be reduced or discontinued from sites that have achieved self- sustainability.

EOWS Encourages Self- Sustainability, yet Needs Criteria to Make Funding Decisions

In 1999, EOWS established a rule to encourage self-sustainability that was generally intended to limit Weed and Seed funding to a site to 5 calendar years and require grantees to shift the majority of EOWS funds to a different site after 5 years. According to EOWS officials, EOWS also encouraged sites to become self-sustaining by establishing activities to be completed during each year of the 5-year grant. Program monitors are to assess sites’ completion of these activities by recording results on a checklist.

We found that while EOWS undertook efforts to encourage self- sustainability, it has not fully developed criteria to make funding decisions based on whether sites had achieved self-sustainability. In fiscal year 2002, 73 sites, which had completed 5 years of EOWS funding, reapplied for funding. EOWS approved an additional round of 5-year funding to 67 of these sites. EOWS denied funding to 6 sites, but not for reasons related to self-sustainability. At the time of our review, no site’s funding had been reduced or withdrawn because the site had achieved self-sustainability, even though EOWS has funded some sites since the early 1990s. In relation to the activities EOWS set to encourage sites to achieve self-sustainability, because these activities were established in 2003, we could not assess the sites’ progress toward completing them. As we reported in 1999, without criteria, EOWS does not have a basis for determining when sites are self- sustaining and when to reduce or withdraw Weed and Seed funds.

EOWS officials told us that OJP is currently developing criteria for self- sustainability for EOWS and other OJP programs and that a report detailing the criteria may be completed in 2004. According to Justice, EOWS developed a new criterion for self-sustainability in the FY2003 Competitive Application Kit regarding whether or not a full-time coordinator is funded by a reallocation of resources other than the Weed and Seed grant.

For information about proposed legislation that could affect self- sustainability for Weed and Seed sites, similar to other Justice programs, see appendix III.

Performance Measures Generally Did Not Track Program Outcomes

In 1999, we reported that while EOWS had developed various performance measures in an attempt to respond to the Government Performance and Results Act (GPRA) of 1993, EOWS’s measures generally did not track program outcomes. GPRA seeks to shift the focus of federal management and decision making away from activities performed to outcomes, or the results of activities undertaken. Since our last report, EOWS has continued to refine and develop its performance measures, but we found that EOWS still generally collects site activity data rather than measuring outcomes. EOWS also generally does not use intermediate measures, which represent conditions believed to precede or contribute to achieving the ultimate outcomes that may be considered constructive steps toward measuring outcomes. While assessing success using outcomes and/or intermediate measures can be difficult, doing so is important to program management and to policy makers for understanding whether the program is achieving the intended results, and for identifying opportunities for improvement.

EOWS Generally Collects Activity Data Rather Than Outcome Performance Measures

Our review showed that EOWS generally collects activity data, but has not developed outcome or intermediate measures that enable EOWS management to track the success of the Weed and Seed program in meeting its goals. As established in prior work by GAO, outcome measures help officials track the success of their programs in meeting program goals. EOWS mostly collects data about site activities, such as whether sites have foot patrols, safe havens, and provide job training. An intermediate measure, for example in relation to job training programs, might be the extent to which those who attend job training obtain employment. This intermediate measure rests on the assumption that individuals who are employed are less likely to commit crimes. See appendix V for activity data collected by EOWS.

EOWS officials told us that to measure its success in reducing violent crime, they collect data on the number of homicides and consider homicides the significant indicator for measuring performance outcomes. However, using the number of homicides as an outcome measure indicative of program success is problematic for several reasons. First, because homicides are relatively rare, even in high crime areas, homicide trends may be too unstable (fluctuate too much from year to year) to assess the success of the Weed and Seed program. Second, outcome measures, such as homicides, can present some methodological challenges because it is difficult to draw a direct causal link between the homicides and a program’s work. Economic trends and other law enforcement initiatives could also be responsible for the observed outcomes. If homicides are used as a performance measure, any analysis should attempt to control for other factors influencing the outcomes. Third, given the broad nature of this program goal -- to reduce violent crime, measuring outcomes in relation to only one type of violent crime also seems problematic because it is too narrowly focused. According to EOWS officials, homicide data was selected because of its reliability, and because gang-related homicides are a significant indicator of the success of a Weed and Seed strategy.

In February 2004,the Office of Management and Budget (OMB) issued a detailed assessment of the Weed and Seed program. According to OMB, Weed and Seed has not demonstrated results. OMB’s report states that Weed and Seed lacks clear targets or timelines for reducing violent crime. Additionally, the report states that Weed and Seed has difficulty collecting data and developing aggregate measures for assessing program performance.

GAO Has Previously Reported on How Other Federal Programs Have Developed Effective Performance Measures, Despite Challenges

Although the Weed and Seed program faces many challenges in developing outcome measures for its various activities, we have previously reported that other federal programs have developed effective performance measures even under difficult circumstances. These reports identified a variety of strategies that other federal programs have used to develop performance measures. For example, we reported that some federal programs utilized a mix of outcome and intermediate measures. This combination of measures allowed them to minimize the risk of not showing outcomes because of their limited control over external factors such as economic trends, which may prevent programs from achieving intended outcomes. Intermediate measures are also helpful to show progress when it is expected to take many years before the desired outcome is likely to be achieved. In such instances, progress toward program outcomes may be demonstrated through intermediate outcomes.

We also reported that where measuring outcomes is, after careful consideration, deemed infeasible, intermediate measures can be used to track progress toward outcomes for programs such as Weed and Seed.

In addition, we have reported other ways federal agencies have used intermediate outcomes. For example, the National Highway Traffic Safety Administration uses the rate of safety belt use as an intermediate measure of its goal to reduce motor vehicle crashes and the resulting fatalities and injuries. As we reported, an increase in the use of safety belts is considered an intermediate outcome--not an end outcome--because it is desirable not in itself but because it has been demonstrated to contribute to the ultimate goal--reducing highway-related fatalities and injuries.

Additional Performance Measures Are Being Developed for EOWS

The Justice Research and Statistics Association and the Urban Institute are currently developing additional outcome performance measures under contracts awarded by EOWS in 2002 and 2003. According to EOWS officials, these research organizations have the expertise needed to develop outcome performance measures for the Weed and Seed program. We interviewed researchers from these organizations and spoke to EOWS officials about their research plans. However, none of the studies on developing additional performance measures had been completed at the time of our review, and it is too soon to tell whether the studies will produce outcome measures needed to adequately assess the Weed and Seed program. The Justice Research Statistics Association and other researchers were conducting studies on crime data and evaluation during our review; one of those studies was submitted to EOWS in November 2003. See appendix VI for a description of EOWS’s ongoing and completed studies.

Conclusions

We believe that 5 years has been ample time for OJP and EOWS to fully implement the recommendations we made in our 1999 report. EOWS has developed and partially implemented policies requiring the completion of documentation to support major decisions, such as qualification and funding decisions, and recorded monitoring information. But by failing to ensure that these documents are appropriately maintained in official recognition and grant files, the documents are not readily available to meet their purpose. That is, the intent of this type of internal control is to ensure that both management and external reviewers such as auditors can adequately perform their responsibilities, for example, reviewing work and making management decisions based on complete and accurate information. While EOWS was eventually able to produce the documentation we requested, failure to appropriately file the documentation made the process of using it very inefficient and, in effect, may have defeated the purpose of having it completed.

While EOWS may have moved forward in addressing its self-sustainability goals by recently developing an activity checklist for grantees to use to document actions that are intended to achieve self-sustainability, the activities do not constitute criteria for determining when sites should be considered self-sustaining and consequently have federal funds reduced or discontinued. One of the benefits of such criteria is to enable EOWS to determine when current projects are likely to be able to self-sustain so that available funding can be used to help reduce crime and achieve other benefits in other deserving communities. Because no sites have had their funding withdrawn because they were deemed self-sustaining during the 13-year life of this program, EOWS may be foregoing the opportunity to use the funds in another location where the need for federal funding is greater or EOWS may need less overall funding. Without criteria to determine when federal funds are no longer needed, EOWS also runs the risk of providing funds beyond what is needed to sustain some sites.

EOWS also needs performance measures that focus on program outcomes, and/or achievement of intermediate goals, so that it and those that provide oversight, such as the Congress and OMB, will be able to adequately assess the extent to which the program is achieving its goals. Management depends on this type of assessment to make the strategic and operations decisions needed to achieve the program’s missions and goals. Congress and oversight agencies need this type of assessment so they can make funding decisions and help ensure that EOWS is in the best position and has the best tools to accomplish its mission. While EOWS recognizes the need for outcome performance measures and has funded studies to help develop them, it is unclear at this time whether these studies will be able to suggest the needed measures.

Recommendations for Executive Action

We recommend that the Attorney General of the United States require the Assistant Attorney General for OJP to ensure that the Executive Office for Weed and Seed fully implement the intent of our previous recommendations by taking the following four steps: maintain the documentation of the basis and rationale for qualification and funding decisions in appropriate grant files; retain progress reports and site visit reports in official grant files; clearly define criteria to assess when sites are self-sustaining and apply the criteria to sites when making further funding decisions; and develop outcome performance measures--or, where measuring outcome is, after careful consideration, deemed infeasible, intermediate measures--that can be used to adequately track progress toward program outcomes of the Weed and Seed program.

Agency Comments and Our Evaluation

We provided a draft of this report to the Attorney General of the United States for review and comment. In a March 11, 2004, letter, the Assistant Attorney General (AAG) for OJP commented on the draft. Her written comments are summarized below and presented in their entirety in appendix VII. Based on Justice’s comments, we modified the text of our report where appropriate and also incorporated Justice’s technical comments into this report where appropriate.

In the AAG’s response, the Justice Department generally agreed with our recommendations on maintaining the documentation of the basis and rationale for qualification and funding decisions in appropriate grant files and retaining progress reports and site visit reports in official grant files. Justice agreed in part with our recommendation on self-sustainability criteria and fully agreed with our recommendation on outcome performance measures, and offered additional information to show it had moved forward in responding to our past recommendation.

Justice agreed that controls should be strengthened to ensure that the basis and rationale for qualification and funding decisions are documented in the centrally maintained official recognition (OR) files. In Justice’s comments, the AAG explained that electronic documents maintained in its grant management system (GMS) are considered a part of the official recognition file and are readily available to EOWS, and that they should have been, but were not, readily available to, and easily accessible by, GAO. As of the current fiscal year, OJP will not only post all solicitations and receive all grant applications via GMS, but also receive and maintain all grant progress reports on GMS. In this way, there will be no opportunity to misplace official grant documents. Justice stated that OJP’s Office of the Comptroller will implement procedures to ensure that external reviewers have ready access to GMS information.

Justice agreed with our second recommendation on retaining progress reports and site visit reports in official grant files. Justice noted that EOWS recently implemented a progress reporting module in GMS that now enables recipients of grants awarded through GMS to submit semi annual progress reports electronically. OJP is developing a monitoring module that will enable program monitors to record and report on site visits directly in GMS. In the interim, EOWS and the Office of the Comptroller will take steps to strengthen controls to ensure that copies of completed site visit reports are included in the official grant file.

Justice agreed in part with our third recommendation on self-sustainability and noted that EOWS will further define the criteria to assess when sites are self-sustaining and apply the criteria when making funding decisions. Further, Justice said it is promoting a “graduation” process to bring sites to sustainability and cease providing Weed and Seed funding to those sites, thus making funding available to assist newer developing sites. However, Justice does not agree that EOWS has never set criteria for self- sustainability. Justice said that EOWS set a criterion for self-sustainability in the FY2003 Weed and Seed Competitive Application Kit, which asked if the application provided for a full-time coordinator funded by reallocation of existing resources other than the Weed and Seed grant. According to Justice, applications were ranked against this criterion. According to EOWS officials, in March 2004, EOWS will announce an additional criterion specific to measuring self-sustainability. The new criterion will require grant applicants to identify other funding sources at a level five times the EOWS contribution. This criterion will be considered when making funding decisions.

We applaud EOWS’s commitment to further develop criteria to assess when sites are self-sustaining and apply the criteria to sites when making further funding decisions. However, Justice’s comments did not provide specific information about how the Application Kit criterion has been used to determine self-sustainability. We added Justice’s belief that this criterion relates to self-sustainability to the text, but also note that over the 13-year history of the program, funding has never been reduced or withdrawn from a site because a site was deemed to be self-sustaining.

Justice agrees that developing outcome performance measures is important and states that EOWS has been working to develop measures that track progress toward program outcomes in the Weed and Seed Program. In a letter commenting on this draft report, Justice provided a summary of results from JRSA’s study to support EOWS’s use of the change in homicides as a measure of program success. According to Justice’s summary, this study concluded that measuring reductions of homicides is an adequate performance measure of the Weed and Seed program because of its reliability and explicit selection as a goal by over 30 percent of the Weed and Seed sites participating in the Crime Pattern Study. In addition, Justice includes a 5 percent reduction in homicides in Weed and Seed sites as part of its Strategic Plan. With its comments, Justice included summaries of two recent research reports. One of these, Analysis of Homicide in Weed and Seed Sites, showed positive results in homicide trends in Weed and Seed sites when compared to host jurisdictions. Another report, Crime Pattern Analysis (Three-Top Crime Study), showed that after the third year of Weed and Seed program implementation, sites encountered a significant decrease in reported crimes.

Because the Crime Pattern Analysis study was not completed at the time of our review, we could not fully assess its methodological rigor and whether it would result in effective outcome performance measures. Although we believe that EOWS may be moving in the right direction regarding developing additional performance measures and we mention the studies in the report text, we also believe that homicides have shortcomings as described in the text when used as the only measure of the program’s performance. In addition, our review of the Analysis of Homicide in Local Weed and Seed Sites study generated several concerns. First, the study used changes in the raw volume of homicides, instead of changes in homicide rates, as its indicator of success or failure of program sites. This method does not account for changes in the population as would be accounted for were a homicide rate measured. Second, the study used the larger host jurisdictions for specific Weed and Seed sites as the comparison locations to which the Weed and Seed sites are compared. In doing so, Weed and Seed sites’ homicide data are included with host jurisdictions’ homicide data, thus making the differences between them extremely difficult to interpret. It is also worth noting that this study evaluates Weed and Seed sites only on the basis of decreases in homicides, while Weed and Seed was intended to decrease crime more generally.

We are sending copies of this report to the Attorney General, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff has any questions on this report, please call Weldon McPhail on (202) 512-8644 or me on (202) 512-8777.

Appendix I: Scope and Methodology

To determine what actions the Executive Office for Weed and Seed (EOWS) has taken to develop an adequate internal control to ensure full documentation of the basis and rationale for qualifying new and existing sites for funding, we reviewed (1) the criteria used to determine which new and existing sites should be qualified for funding, and (2) EOWS’s policies and guidance. To gather this information, we interviewed officials from Justice and EOWS and reviewed pertinent documents, including EOWS policies and procedures, official recognition and grant applications, management oversight and monitoring documentation, and budget reports. In addition, we randomly selected 10 of 94 fiscal year 2002 official recognition files submitted to EOWS for review. These 10 files included 5 files from new applicants and 5 files from existing sites that applied in 2002 and were funded in fiscal year 2002. We reviewed 10 of the 14 official recognition files that EOWS decided not to fund in fiscal year 2002. We also reviewed 10 randomly selected official grant files from the 31 new and existing sites that were approved for funding in fiscal year 2002. Because of limited resources, we did not attempt to review all the files. While the small sample size prevents us from making reliable generalizations, the 30 files we reviewed represented various types of files that EOWS maintains. We selected the files from a list that was provided to us by EOWS. To ensure the list of files was sufficient for the purpose of our review, we spoke to knowledgeable EOWS officials about the completeness and accuracy of the list.

To assess what steps EOWS has taken to improve program monitoring to ensure that sites meet the grant requirement of submitting progress reports, and that EOWS site visits are documented, we reviewed EOWS program grant guidance, including training offered, the EOWS monitoring guidance used by grant monitors when conducting site visits, and the grant files for the Weed and Seed sites that we visited: Pittsburgh, Pennsylvania; Charleston, South Carolina; and North Charleston, South Carolina. We selected these sites from the 221 sites funded by EOWS in fiscal year 2003. Based on our discussions with EOWS officials about the sites, we selected three sites, which although not representative of all sites, had received Weed and Seed grant funds since the early 1990s, which enabled us to learn about their efforts to implement the Weed and Seed strategy over time. We also reviewed selected monitoring visit reports prepared by grant monitors for these sites and biannual progress reports submitted in fiscal years 2002 and 2003. We interviewed EOWS officials and grant coordinators at these 3 sites regarding procedures used for monitoring Weed and Seed sites. We also reviewed the same 10 randomly selected official grant files discussed above from the 31 new and existing sites that were funded in fiscal year 2002 to determine whether the progress reports and site visit reports were in the official grant files and readily available. The sites were funded under EOWS’s competitive application and did not include sites funded under EOWS’s continuation application. While the 10 files we reviewed represented various types of files that EOWS maintains, the small sample size prevents us from making reliable generalizations about all official grant files. To ensure the list of files was sufficient for the purpose of our review, we spoke to a knowledgeable official about the completeness and accuracy of the list.

To assess what criteria, if any, EOWS has developed when sites are self- sustaining and when to reduce or withdraw program funding, we asked EOWS officials for any relevant information. Following repeated requests, EOWS officials did not provide any documentation on criteria that they had developed. They did, however, provide information on the five-year rule for self-sustainability and the sites’ activities, which we reviewed.

To determine what additional performance measures EOWS has developed that track program outcomes, we interviewed officials from EOWS and the sites we visited. We reviewed pertinent documents, including EOWS policies and procedures, grant applications, and data collected pursuant to the Government Performance and Results Act of 1993 (GPRA). In relation to the studies EOWS commissioned to develop program-wide performance measures, we interviewed the authors of the studies under way to discuss their research plans. At the time of our review, none of the studies on developing additional performance measures had been completed.

We conducted our audit work between June and December 2003 in accordance with generally accepted government auditing standards.

Appendix II: GAO Site Visit Summary

Pittsburgh, Pa.

Pittsburgh established its first Weed and Seed site, the Hill District, in 1992. In 1995, Hazelwood became the second Pittsburgh site. In 1997 and 1999, East Liberty and Homewood became the third and fourth Weed and Seed sites in Pittsburgh. Pittsburgh currently receives funds for three sites: East Liberty, Hazelwood, and Homewood. The sites are located between the Allegheny and Monongahela rivers on the eastern side of the city. The population of the target area is 10,897 in East Liberty, 15,849 in Hazelwood, and 9,283 in Homewood. In fiscal year 2003, the Pittsburgh sites in total received $435,000 in Weed and Seed funding. See table 2 for the Weed and Seed funding history for Pittsburgh’s three sites.

Pittsburgh’s weeding activities include funding a drug violence initiative headed by the Federal Bureau of Investigation (FBI) and a community- based crime prevention activity coordinated with the police. Pittsburgh’s seeding activities include funding a drug education youth camp that provides children with leadership training and fitness tests, and a downtown revitalization program. See table 3 for examples of the types of activities funded by the Pittsburgh Weed and Seed program.

An important goal of the Weed and Seed program is for sites to leverage resources from sources other than EOWS in order to become self- sustaining. During our site visit and through documentation obtained from Pittsburgh Weed and Seed program staff, we identified several partnerships established by the Pittsburgh Weed and Seed program to leverage resources. These cooperative arrangements involved partners such as the University of Pittsburgh Medical Center and the Pittsburgh Harlequin Rugby Association. Table 4 provides specific examples of leveraging efforts that we identified. Pittsburgh Weed and Seed program staff estimate that they coordinated and leveraged approximately $1.1 million in 2002.

Charleston, S.C.

Charleston became a Weed and Seed site in the spring of 1992. The Charleston Weed and Seed target area includes approximately 8 square miles on the Charleston Peninsula and in the city of Charleston. The entire Weed and Seed site is located in Charleston’s Renewal Community. The population of the target area, encompassing 13 neighborhoods, is roughly 17,000. In fiscal year 2003, the site received $275,000 in Weed and Seed funding. See table 2 for Charleston’s Weed and Seed funding history.

Charleston’s weeding activities include supporting a youth court with a Weed and Seed juvenile investigator and funding community policing foot patrols. Charleston’s seeding activities include supporting a police-youth after-school boxing program, and boarding up abandoned and vacant buildings to improve the appearance of the area. See table 3 for examples of the types of activities funded by the Charleston Weed and Seed program.

An important goal of the Weed and Seed program is for sites to leverage resources from sources other than EOWS in order to become self- sustaining. During our site visit and through documentation obtained from Charleston, we identified several partnerships established by the Charleston Weed and Seed program to leverage additional resources. These cooperative arrangements involved partners such as the FBI and the College of Charleston’s School of Education. Table 4 provides specific examples of leveraging efforts that we identified. Charleston Weed and Seed program staff estimate that they were able to coordinate and leverage approximately $1 million in 2002.

North Charleston, S.C.

North Charleston was originally funded as a Weed and Seed site in August 1993. The city of North Charleston is located about 7 miles north of the city of Charleston. Today, North Charleston has four target neighborhoods: Union Heights, Chicora/Cherokee, Accabee, and Liberty Hill. The population of the target area is 13,606. In fiscal year 2003, the site received $225,000 in Weed and Seed funding. See table 2 for North Charleston’s Weed and Seed funding history.

North Charleston’s weeding activities include participating in Operation Cease Fire with the U.S. Attorney’s Office, to reduce gun crime in target neighborhoods, and partnering with local law enforcement to improve the bike patrol program. North Charleston’s seeding activities include developing the Drug Education for Youth program and supporting an annual neighborhood cleanup. See table 3 for examples of the types of activities funded by the North Charleston Weed and Seed program.

An important goal of the Weed and Seed program is for sites to leverage resources from sources other than EOWS in order to become self- sustaining. During our site visit and through documentation obtained from North Charleston, we identified several partnerships established by the North Charleston Weed and Seed program to leverage resources. These cooperative arrangements involved partners such as the United States Air Force and the Medical University of South Carolina. Table 4 provides specific examples of leveraging efforts that we identified. North Charleston Weed and Seed program staff estimate that they coordinated and leveraged approximately $3.3 million in 2002.

Appendix III: Proposed Legislation Could Affect Self-Sustainability

Proposed legislation could affect self-sustainability for Weed and Seed sites by mandating that EOWS impose more rigorous requirements on its grantees, similar to some other Justice programs. Unlike some other Justice discretionary grant programs, the law establishing the Weed and Seed program currently does not have a matching funding requirement, nor does it limit the maximum number of years a grantee can receive funding. H.R. 3036, a bill to reauthorize the Department of Justice for fiscal years 2004 through 2006, was introduced on September 9, 2003. This bill includes a provision that would limit the federal contribution to 75 percent of the total Weed and Seed program costs. The bill would also impose limitations on the duration of grants funded under the Weed and Seed program, limiting to 10 the total number of fiscal years a grantee may receive grants. Additionally, the bill requires that in order for a grantee to be eligible for a grant, it must agree to formulate a timely and effective plan to independently sustain the Weed and Seed program when federal funding ends.

Currently there are other federal grant programs that require a funding match and a limit on the duration of grant awards. For example, the Drug- Free Communities Support Program (DFCSP) and Community Oriented Policing Services (COPS) both have matching requirements and clearly specify the number of years a grantee may be funded. DFCSP is a discretionary grant program with a budget of about $60 million in fiscal year 2003. DFCSP requires a dollar-for-dollar match, limits grantees to no more than 5 years of funding, and reduces awards by 25 percent in the fourth and fifth years. The COPS program, with a budget of about $929 million in fiscal year 2003, generally requires grantees to contribute 25 percent of the costs of the program. The federal share of a COPS grant that covers more than 1 year must decline year to year.

Appendix IV: Executive Office for Weed and Seed Selected Activities

The steering committee continually educates, guides, and reevaluates site management.

Steering committee meets at least monthly.

Steering committee and organizational structure continue to function. Steering committee meets at least quarterly.

The grantee and/or local U.S. Attorney’s Office evaluates the site coordinator’s job performance, with input from the steering committee.

The site coordinator and the steering committee work together to ensure timely expenditure of grant funds.

The site’s strategy to achieve measurable outcomes in the four required program areas (especially law enforcement/community policing) is on track. The site continues to collect data related to goals and objectives.

The site continues to implement strategy for achieving goals and objectives in all four required program elements.

The site makes plans to continue community development efforts and strategic planning through various funding sources.

The site continues program publicity and outreach to recruit neighborhood leaders and mobilize community.

The site continues program publicity and outreach efforts to recruit neighborhood leaders and to mobilize the community while identifying and building sustainable leadership among community members.

The site continues program publicity and outreach efforts to mobilize community and recruit new community leaders.

The site’s community leaders are working to sustain efforts and continue community mobilization.

The site continues to coordinate its efforts with local initiatives, particularly programs with local parks and recreation departments.

The Weed and Seed effort begins to be included in city/community plans.

Community leaders have established working relationships with local officials.

The Weed and Seed community is included in city/community planning.

The steering committee develops and implements the special emphasis program based on community needs.

The special emphasis program is fully operational.

The site submits a final progress report.

The site communicates regularly with all partners, including U.S. Attorney’s Office, Executive Office for Weed and Seed, and community members, using newsletters, faxes, neighborhood visits, e-mail, community meetings, and so forth.

The steering committee and site coordinator review previous years’ grant applications to determine plans for final year of official recognition.

The steering committee determines if there is a need to continue the program in other neighborhoods based on the program.

The site establishes a subcommittee to look for additional funding or in-kind resources to sustain the program.

Subcommittee continues to search for additional resources to sustain the program.

The site begins to develop plans for evaluating its programs.

The site identifies an evaluation tool and begins developing an evaluation based on Executive Office for Weed and Seed site evaluation literature.

The site undergoes program evaluation to assess overall effectiveness of official recognition strategy implementation.

By the end of the year, the site releases its full evaluation report, assessing the overall program accomplishments and its effectiveness.

A site is a geographically defined area ranging in size from several neighborhood blocks to several square miles. With the input of the U.S. Attorney’s Office and the steering committee, each site develops and implements its own Weed and Seed strategy.

Official recognition is a designation given to a grantee by EOWS that signifies that EOWS has approved the grantee’s Weed and Seed strategy in a specific site. A grantee must receive official recognition from EOWS before a grantee can apply for funding. Official recognition requires, among other things, that the site develop a management plan and a comprehensive strategy for implementing the Weed and Seed strategy.

The four required program elements that sites are required to address are (1) law enforcement; (2) community policing; (3) crime and substance prevention, intervention, and treatment; and (4) neighborhood restoration.

Appendix V: Activity Data Collected by EOWS in Fiscal Year 2003

Appendix VI: Description of EOWS’s Ongoing and Completed Studies

Compare homicides in Weed and Seed sites with homicides in the jurisdictions that sites are located in, to determine whether homicides will decline faster in Weed and Seed sites than in the remainder of the jurisdiction.

Develop additional performance indicators that will be used by EOWS and Weed and Seed sites to evaluate program performance with respect to crime control.

Summarize the types of crime selected as “target” crimes by Weed and Seed sites (sites choose three crimes) to determine crime reduction strategies. Assess the degree of success of Weed and Seed sites by analyzing the percentage of reduction in crime compared with the percentage in the rest of the jurisdiction that they are located in.

Summarize the completed evaluations of individual Weed and Seed sites. The analysis will include more than 80 studies. JRSA’s preliminary analysis (March 3, 2003) includes 36 evaluations. Of the 36 evaluations, JRSA determined that 5 were process, 19 were impact, and 9 were both process and impact evaluations. The remaining 3 evaluations were not identified as process or impact.

Conduct local evaluations of Weed and Seed sites. In fiscal year 2002, EOWS funded 13 local evaluations. Each site contracts with its own researcher.

Appendix VII: Comments from the Department of Justice

Appendix VIII: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, the following individuals made key contributions to this report: Lisa G. Shibata, Kirstin B. L. Nelson, Cady L. Summers, David Alexander, Sidney Schwartz, Kevin L. Jackson, Elizabeth Curda, Tom James, Denise Fantone, Jan Montgomery, Ann H. Finley, Katherine M. Davis, and Leo Barbour.

Related GAO Products

Law Enforcement: Better Performance Measures Needed to Assess Results of Justice’s Office of Science and Technology, GAO-04-198. Washington, D.C.: November 14, 2003.

Standards for Internal Control in the Federal Government. GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999.

Federal Grants: More Can Be Done to Improve Weed and Seed Program Management. GAO/GGD-99-110. Washington, D.C.: July 16, 1999.

Small Business Administration: Enhancements Needed for Loan Monitoring System Benchmark Study. GAO/AIMD-99-165. Washington, D.C.: May 14, 1999.

Managing for Results: Measuring Program Results That Are Under Limited Federal Control. GAO/GGD-99-16. Washington, D.C.: December 11, 1998.

Managing for Results: Analytic Challenges in Measuring Performance. GAO/HEHS/GGD-97-138. Washington, D.C.: May 30, 1997.

Federal Grants: Design Improvements Could Help Federal Resources Go Further. GAO/AIMD-97-7. Washington, D.C.: December 18, 1996.

Highway Safety: Safety Belts Use Laws Save Lives and Reduce Costs to Society. GAO/RCED-92-106. Washington, D.C.: May 1992.