Mental Health: Better Documentation Needed to Oversee Substance Abuse and Mental Health Services Administration Grantees
Released: 2015-06-11T00:00:00

Letter

Background

Grants Awarded by CMHS

CMHS awards two types of grants—formula grants and discretionary grants—to support mental health programs. According to CMHS, in fiscal year 2013, about 61 percent of CMHS’s $822 million in grant funding was awarded through formula grants and about 39 percent was awarded through discretionary grants. (See fig. 1.)

Formula grants are awarded to eligible grantees that meet specified criteria outlined in formulas prescribed by statute or regulation. These formulas may consider factors such as the population-at-risk and cost of services. These grants are generally awarded to states and territories that distribute funds to various governmental or nongovernmental entities, such as a state mental health agency or a community mental health center. In fiscal year 2013, CMHS awarded about $501 million under three formula grant programs: 1. the MHBG program, which supports adults with serious mental illness (SMI) or children with serious emotional disturbances (SED); 2. the PAIMI program, which supports protection and advocacy systems designed to ensure the rights of individuals with mental illness who are at risk for abuse and neglect; and 3. the Projects for Assistance in Transition from Homelessness program, which supports outreach, mental health, and other support services to homeless people with SMI.

Discretionary grants are generally awarded on a competitive basis for specified projects that meet statutory eligibility and program requirements. Discretionary grants allow CMHS to allocate funding to a particular issue, such as suicide prevention, or to areas and populations with the greatest need. CMHS discretionary grants may be awarded to state, local, territorial, and tribal governments; institutions of higher education; other non-profit organizations (including tribal organizations); and hospitals. These grant applications are solicited through RFAs that specify the purpose of the grant, eligibility requirements, and grantee reporting requirements throughout the grant’s project period. The duration of a grant’s project period and the amount of funding available each year differ by grant program based on program requirements or statutory requirements. In fiscal year 2013, CMHS awarded 589 discretionary grants totaling about $321 million. The smallest grant award was about $30,000 and the largest grant award was about $6 million.

Grant Review and Award Process

CMHS uses separate processes for reviewing and awarding grants for the formula and discretionary grant programs from which we drew our selection; figures 2 and 3 illustrate these two processes.

Evidence-Based Practices

After CMHS awards grants, grantees can use a variety of different approaches to treat individuals with mental illness, including evidence- based practices. Since 1997, SAMHSA has sponsored a National Registry of Evidence-Based Programs and Practices, which is a searchable online registry of programs and services that are considered by SAMHSA to be evidence-based. According to SAMHSA the purpose of this system is to help the public, including grantees that have been awarded grants by CMHS, learn more about available evidence-based practices.

CMHS’s Criteria for Awarding Grants to Grantees Varied by Program, but CMHS Did Not Document Its Application of Criteria for About a Third of the Grantees We Reviewed

The criteria CMHS established for awarding grants to grantees for the MHBG, PAIMI, and selected discretionary grant programs varied by program. CMHS did not document its application of criteria when awarding grants to grantees for about a third of the grantees we reviewed.

CMHS’s Criteria for Awarding Grants to Grantees for the MHBG, PAIMI, and Selected Discretionary Grant Programs Varied by Program

The criteria CMHS established for awarding grants to grantees for the MHBG, PAIMI, and selected discretionary grant programs varied by program. These criteria identify the requirements grantees must meet in order to receive a grant, including any requirements identified in statute, regulation, or the terms and conditions for the grant program, such as those outlined in RFAs. For example, one of the five programs that awarded grants to grantees we reviewed—Project LAUNCH—required its grantees to state that they will use evidence-based practices to treat individuals with mental illness in their applications when such practices are available, while the others did not. See table 2 for examples of CMHS’s criteria for awarding grants to grantees.

CMHS Did Not Document Its Application of Criteria for About a Third of the Grantees We Reviewed and CMHS Lacked Program-Specific Guidance for How to Document This Information

During the 2-year period covered by our review, CMHS did not document its application of the criteria it used to award grants to 6 of the 16 grantees we reviewed. Specifically, we found instances across the 2 years in which CMHS did not clearly document the application of its criteria for 4 MHBG grantees and 2 PAIMI grantees. We found that CMHS documented the criteria it applied when awarding grants to discretionary grantees. The grants manual states that CMHS must maintain appropriate file documentation to support decisions in the financial assistance process, including funding decisions, and that transactions and significant events are to be clearly documented to help management with decision-making to help ensure operations are carried out as intended. In addition, Standards for Internal Control in the Federal Government state that all transactions and other significant events need to be clearly documented. CMHS officials said that they follow the grants manual but said that they do not have written guidance that is specific to the MHBG and PAIMI programs that would assist project officers in using the tools that CMHS has developed, such as checklists, to document their application of criteria. Examples of instances in which CMHS did not document its application of criteria when awarding grants to the grantees we reviewed include the following:

MHBG. For fiscal year 2012, CMHS officials did not clearly document the application of most criteria for any of the four MHBG grantees we reviewed; however, officials did document how they applied most of the criteria for fiscal year 2013. Because fiscal year 2013 was the second year of the grant project period that corresponded to the application for continued funding, it required that CMHS apply fewer criteria than in the initial application for fiscal year 2012. Specifically, CMHS officials use checklists to help them determine whether grantees meet the agency’s criteria when deciding whether to award grants to grantees. Grantees are expected to provide sufficient information to demonstrate that they have satisfied the criteria in their grant applications. However, for all four of the grantees we reviewed, the checklists for the application covering fiscal year 2012 data listed “N/A” or had missing responses for most of the items in the checklist—the majority of which were related to criteria identified in statute. One grantee’s checklist we reviewed listed “N/A” or had missing responses for nearly all of the items related to statutory requirements. For example, all items related to the five criteria that must be addressed by the required plan submitted as part of the application were listed as “N/A”, and none of the items related to the required state priorities were completed by the project officer. One project officer told us that if a required item on the checklist is listed as “N/A”, it is because the project officer did not have the specified information from the grantee and would therefore request the information. However, we did not see evidence that the project officer updated the checklists after receiving additional information. In another case, officials told us that there was a technical problem with the system that electronically stores the checklists and the responses were automatically populated with “N/A” for these items. While we found that none of the checklists for the MHBG applications covering fiscal year 2012 data clearly documented how CMHS applied criteria when reviewing applications, we found that the checklists covering fiscal year 2013 data for the four grantees clearly documented the use of most criteria. However, officials explained that the 2013 application and the corresponding checklist was less extensive because it was an application for continued funding; project officers reviewed 19 criteria for the 2013 checklist compared to 129 for 2012.

PAIMI. For fiscal year 2012, CMHS officials documented the application of criteria for three of the four PAIMI grantees we reviewed. However, we were unable to determine whether CMHS applied the criteria for the fourth grantee because CMHS officials were not able to provide the checklist they used to review this grantee. For fiscal year 2013, we found that accurate documentation of the application of criteria was not available for 2 of the 4 grantees we reviewed. Similar to the MHBG program, officials use checklists to help them determine whether grantees met CMHS’s criteria, such as whether the grantee demonstrated that it had identified priorities and objectives that were relevant to the PAIMI program. All four of the grantees’ checklists for the fiscal year 2013 applications contained documentation that indicated that the project officer found that the objectives identified by the grantee were not relevant to the PAIMI program. However, CMHS officials said that this finding was an error for two of the grantees, but was correct for the other two grantees. The officials told us that, as of October 2014, they had not conducted any follow-up with the two grantees to request additional information regarding the noted concerns. Officials told us that they would only follow up with the grantees on the noted concerns if they conducted a site visit.

We found that there were a variety of reasons why CMHS did not adequately document the application of criteria when awarding grants to grantees. One reason was that officials told us they lacked program- specific guidance to document the application of criteria for some programs. CMHS has developed tools for project officers to use for their application of criteria, but project officers sometimes had different ideas about how to use those tools and what to do if a grantee could not provide needed information. For example, officials told us that for the MHBG program there was no written guidance, during the period covered by our review, that described how project officers were to complete the application review checklists to help ensure grantees met the grant criteria. Officials developed program-specific guidance for project officers in fiscal year 2014 for the review of MHBG applications, but we did not assess whether this guidance fully addressed the issues we identified. CMHS officials told us that they also provide training to project officers and that they believe the process is clear; however, we found that project officers had different understandings of how to complete the checklists that are intended to document their review of criteria. Officials also noted a technical problem in the system used to document the application review checklists in fiscal year 2012. For the PAIMI program, officials told us that there was no formal process to address instances in which a grantee did not meet criteria, including how to follow-up with the grantee, for the period we reviewed. Project officers told us that they would sometimes note concerns in the checklist or on the applications, but these notes were for their own benefit. Further, they told us that they generally do not follow-up with the grantee unless they have a site visit. Without documentation that clearly identifies how criteria have been applied when awarding grants to grantees, CMHS cannot ensure that it is applying criteria consistently and appropriately.

CMHS Stated That It Uses Various Types of Information for Oversight, but the Documentation of This Information Was Often Missing or Not Readily Available

CMHS officials said they use various types of information to oversee grantees awarded grants through the MHBG, PAIMI, and selected discretionary grant programs. However, we found at least one instance during the period covered by our review where documentation of this information was either missing or not readily available for each grantee we reviewed.

CMHS Officials Said They Use Various Types of Information, Such as Financial Information and Progress Reports, to Oversee Grantees

CMHS officials said they use various types of information to oversee grantees awarded grants through the MHBG, PAIMI, and selected discretionary grant programs. The grants manual, which CMHS officials told us they use as guidance for grantee oversight, describes the various types of information that should be used for overseeing grantees. This includes information grantees are required to report during and at the conclusion of the grant, such as financial information and progress reports, and other information the agency uses for grants management, such as documents noting project officer’s approval of information submitted by the grantee. According to the grants manual, this information is intended to hold the grantee accountable to the programmatic and financial conditions of the award and provide a means for ongoing monitoring of grantee performance. Examples of the types of information and related documentation CMHS officials stated they use to oversee grantees we reviewed for fiscal years 2012 and 2013 are outlined in table 3 below.

Documentation of Some Information CMHS Officials Said They Used to Oversee Grantees Was Either Missing or Not Readily Available for Each of the Grantees We Reviewed and CMHS Lacked Program-Specific Guidance

For each of the 16 grantees we reviewed, we found at least one instance during the period covered by our review in which the documentation CMHS officials said they used to oversee grantees was missing or not readily available—meaning it was either missing entirely, stored outside of the systems CMHS designated for storing the information, or was not readily available to all officials involved in the oversight of grant documentation. Because this information was missing or not readily available, we were unable to identify all of the information project officers used to oversee these grants. The grants manual states that CMHS must create and maintain files that allow a third party, such as an auditor, to “follow the paper trail” beginning with program initiation through closeout of individual awards, including decisions made and actions taken in between. As previously noted, examples of the types of information that should be documented include all information required by the conditions of the award, such as financial and performance reports. According to the grants manual, this documentation should provide a complete record of the history of an award and serve as a means to support funding decisions and provide ongoing monitoring of grantee performance. In addition, Standards for Internal Control in the Federal Government state that all transactions and other significant events need to be clearly documented, and the documentation should be readily available for examination and properly managed and maintained. CMHS officials said that they follow the grants manual. However, where we found documentation missing or not readily available, CMHS often lacked program-specific guidance to assist project officers in documenting their oversight of grantees. Table 4 below provides information on the types of issues identified for the grant documentation for grantees we reviewed from each of the three grant programs.

Specific examples of grant documentation that was missing or not readily available for grantees we reviewed from each of the three grant programs include the following:

MHBG. Financial information grantees are required to report was missing for two of the four MHBG grantees we reviewed because, according to CMHS officials, the grantees did not submit it. We found that one grantee had not submitted final financial expenditures for fiscal years 2012 or 2013. The other grantee had not yet submitted final expenditures for fiscal year 2013. CMHS officials provided documentation that indicated that these grantees subsequently submitted this information after our review was complete. In addition, we found that the same two MHBG grantees had not submitted required information to show that they have maintained a required level of state mental health expenditures. Further, we found that there was no documentation from either of these grantees or from CMHS officials that indicated that a waiver had been approved to relieve the grantee of this requirement or that CMHS had determined that the grantee had materially complied despite not maintaining the required level of expenditures. This documentation is required in instances in which the grantee has not complied with the spending level. These grantees submitted other required financial information after our review was complete and CMHS officials told us that as a result this documentation was no longer needed. In addition, for all four grantees we reviewed, documentation of the final approval by the branch chief for the application covering fiscal year 2012 was not readily available. Specifically, the approval in the application review system was marked pending. According to officials, the application did not appear to have been approved because there were technological issues with the system CMHS maintains to record approvals. As a result, the branch chief provided final approval verbally for these grantees but did not document it.

PAIMI. CMHS could not produce documentation of its review of the required annual program performance reports covering fiscal year 2012 data for any of the four PAIMI grantees we reviewed. Project officers told us that they typically document their review of the reports by writing notes in the margin of a paper copy of the annual report; however, we did not see evidence of any notes in the paper copy reports we reviewed or any other documentation of their review. According to CMHS officials, CMHS developed a checklist in 2013 to help project officers document their review of future annual reports.

We found that CMHS had completed these checklists for all four grantees we reviewed. However, one checklist indicated that the priorities and goals were poorly written and not all of the objectives listed were relevant to the PAIMI program. Officials told us that they did not have any other documentation related to these issues, including any communication with the grantee. In addition, each year officials use data submitted by grantees to calculate aggregate performance measure data across all PAIMI grantees. Officials explained that this information is hand-tallied on paper worksheets. We found that officials did not keep records of the hand-tallied worksheets they used to calculate aggregate performance measure data for fiscal year 2012. Officials told us that they saw no need to retain documentation of these calculations once the data were published in the annual report. Officials also told us that although senior officials review the worksheets for consistency and completeness, they did not maintain any documentation of this review.

Discretionary. For seven of the eight discretionary grantees we reviewed, some performance measure data had not been approved by the project officer in CMHS’s performance measure data tracking system. CMHS officials said that project officers are expected to record their approval of performance measure data in CMHS’s performance measure data tracking system. For the eighth discretionary grantee we reviewed, the grantee had not submitted any performance measure data since the grant was awarded in July 2013. According to CMHS officials, the performance measure data did not appear to have been approved because CMHS’s performance measure tracking system locked, meaning that it did not allow the project officer to enter his or her approval, to enforce project officer deadlines. In some of these cases, project officers used other methods to document their approval of performance measure data, such as hand-written notes. However, according to officials, these notes are generally stored at project officers’ workstations and cannot be easily accessed by other CMHS officials, making it difficult for a third party to follow the trail of project officer oversight and for other officials to access the documents if the project officer is unavailable. In other cases, there was no documentation that the project officer approved this performance measure data.

We found that there were a variety of reasons why documentation used to oversee grantees was missing or not readily available. For example, officials told us that in some cases they lack program-specific guidance for the processes officials use to document their oversight of MHBG and PAIMI grantees. For the MHBG program, officials told us that they did not have program-specific guidance for the period of our review that indicated how project officers and the branch chief are to document their approval of the application if there are technological problems with the application review system. For the PAIMI program, officials told us that they have some program-specific guidance; however, officials told us that there is no written guidance with instructions for how officials are to calculate the PAIMI aggregate performance measure data from the data submitted by grantees. Further, there is no written guidance that describes how this information should be maintained. For its discretionary grant programs, CMHS does have program-specific guidance that indicates how project officers should review some performance measure data submitted by grantees; however, this guidance does not indicate how project officers should approve performance measure data after the system locks. CMHS officials said that some grantees have difficulty meeting the requirements of their grants because they serve high need populations. However, most of the problems we identified were related to documentation that is to be completed by CMHS officials for grants management and not due to issues with the grantees. Without proper documentation of information used to oversee grantees that is readily available, CMHS runs the risk that it does not have complete and accurate information needed to provide sufficient oversight of its grant programs. CMHS officials said that SAMHSA began efforts in fiscal year 2015 to update existing guidance and develop additional guidance for its grant programs, including those administered by CMHS. However, since these efforts are still in early stages, it is too soon to determine whether they will resolve the issues we identified.

CMHS Takes A Variety of Steps When Reviewing Performance Measure Data to Demonstrate How Its Grant Programs Further the Achievement of SAMHSA’s Goals

CMHS officials told us that they take a variety of steps when reviewing grantees’ performance measure data to demonstrate how CMHS’s grant programs furthered the achievement of SAMHSA’s goals, which are identified through the strategic initiatives contained within SAMHSA’s strategic plan. CMHS collects performance measure data from grantees as a way to assess grant program performance. CMHS officials said that their performance measure data generally indicate that CMHS has made progress in achieving SAMHSA’s goals. The data collected is based on performance measures CMHS developed in response to the Government Performance and Results Act of 1993 (GPRA), as amended by the GPRA Modernization Act of 2010. Examples of these measures include the number of evidence-based practices implemented, the number of children screened for mental health or related interventions, and the number of people served by the public mental health system. Grantees provide data to CMHS officials in response to these measures periodically, based on the requirements designated by each grant program. For example, the discretionary grantees we reviewed were required to report performance measure data either biannually or on a quarterly basis, while the MHBG and PAIMI grantees we reviewed were required to report performance measure data on an annual basis. (See app. I for more information on the performance measures for the grantees we reviewed.)

CMHS officials stated that they take a variety of steps when reviewing performance measure data to demonstrate how CMHS grant programs further the achievement of SAMHSA’s goals. First, CMHS officials stated that they review performance measure data reported by grantees for each grant program on an annual basis as part of their budget planning and formulation process. Specifically, CMHS produces summaries by grant program that are included as part of its budget justification. Officials stated that these summaries include tables with performance measure data that demonstrate how the grant programs further the achievement of SAMHSA’s goals. For example, SAMHSA’s fiscal year 2015 budget justification provides information on the number of people served by the public mental health system for the MHBG program who lived in private residences during fiscal year 2012, which CMHS indicated is related to SAMHSA’s goal to ensure that permanent housing and supportive services are available for individuals in recovery from mental health and substance use disorders. Second, some grant programs produced additional reports that demonstrated how these grant programs furthered the achievement of SAMHSA’s goals. For example, CMHS produces a summary report of performance measure data across discretionary grant programs on an annual basis. The report provides graphs with performance measure data across several discretionary grant programs for a given year, and describes how this performance measure data furthers the achievement of SAMHSA’s goals. Third, CMHS officials stated that performance measure data is reviewed by officials assigned to each strategic initiative who report to the administrator of SAMHSA on an ongoing basis to demonstrate the progress of the agency in furthering the achievement of its goals. For example, through this effort, officials stated that they reviewed performance measure data to demonstrate the agency’s progress in achieving the goals within the recovery support strategic initiative and presented these reports to the administrator of SAMHSA.

In addition to these steps, in January 2015, CMHS began its implementation of the Common Data Platform for its discretionary grant programs. This platform is an electronic system that allows officials to generate reports on performance measure data collected from grantees awarded grants through programs across SAMHSA’s centers, including CMHS. According to SAMHSA, analyzing performance measure data across SAMHSA’s centers can assist the agency in evaluating the overall effectiveness of its grant programs and in ensuring that each program furthers the achievement of SAMHSA’s goals. For example, SAMHSA documentation indicates that the Common Data Platform will help officials identify the number of people served by SAMHSA grantees in a particular state and compare that number to previous years’ data as a way of measuring the impact of SAMHSA’s grant programs. While CMHS has begun its implementation of the Common Data Platform for discretionary grant programs, officials stated that CMHS will extend the platform in 2017 to include performance measure data collected from the MHBG and PAIMI programs.

Conclusions

CMHS’s grant programs support services for individuals with mental illness, which is widespread in the United States. Among the grantees we reviewed, we identified concerns with CMHS’s documentation of its application of criteria when awarding grants to grantees and of the information it used to oversee grantees. We found that there were several reasons why documentation was missing or not readily available for the grantees we reviewed. These reasons included a lack of program-specific guidance for the tools and processes that CMHS officials have developed to document the oversight of grantees. While CMHS has developed tools and processes for its staff to use to document key elements of grants oversight, CMHS staff did not always understand how to use them. CMHS has developed some program-specific guidance to help officials oversee grantees and officials stated that SAMHSA began efforts in fiscal year 2015 to update existing guidance and develop additional guidance for CMHS’s grant programs; however, because these efforts are still in early stages, it is too soon to determine whether they will address the issues we found. CMHS officials said that some grantees have difficulty meeting the requirements of their grants because they serve high need populations. However, most of the problems we identified were related to documentation that is to be completed by CMHS officials for grants management and not due to issues with the grantees. Both the grants manual, which CMHS officials said they follow to guide their grant oversight efforts, and Standards for Internal Control in the Federal Government, which apply to all government transactions, state that all transactions and other significant events need to be clearly documented, and that the documentation should be readily available. Without complete documentation of key elements of the oversight of its grant programs, CMHS does not have reasonable assurance that it is overseeing its grant programs effectively to achieve SAMHSA’s goals.

Recommendation for Executive Action

To assure the consistent documentation of the application of criteria to award grants to grantees and of the information used for oversight, the Administrator of SAMHSA should direct CMHS to take steps, such as developing additional program-specific guidance, to ensure that it consistently and completely documents both the application of criteria when awarding grants to grantees, and its ongoing oversight of grantees once grants are awarded.

Agency Comments

We provided a draft of this report to HHS for comment. HHS provided written comments, which are reprinted in appendix II. HHS concurred with our recommendation and stated that the administrator of SAMHSA directed the agency, including CMHS, to initiate efforts to ensure that it consistently and completely documents both the application of criteria when awarding grants to grantees and its ongoing oversight of grantees once grants are awarded. HHS also provided examples of efforts SAMHSA is undertaking to improve the management of its grant programs, including revising and updating guidance used for grants management. However, because several of the efforts are still in development, it is too early to determine whether these efforts will address the issues we identified. In addition, HHS provided technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix III.

Appendix I: Examples of Performance Measures for the Center for Mental Health Services Grantees We Reviewed

This appendix provides additional information on the performance measures that the Center for Mental Health Services (CMHS) developed for the grantees we reviewed from five CMHS grant programs— Community Mental Health Services Block Grant (MHBG), Protection and Advocacy for Individuals with Mental Illness (PAIMI), Systems of Care Expansion Implementation Cooperative Agreements, Statewide Consumer Network, and Cooperative Agreements for Linking Actions for Unmet Needs in Children’s Health (Project LAUNCH). CMHS collects performance measure data from grantees based on these measures periodically as a way to assess grant program performance. CMHS developed its performance measures in response to the Government Performance and Results Act of 1993 (GPRA), as amended. See table 5 for examples of performance measures for the grantees we reviewed.

Appendix II: Comments from the Department of Health and Human Services

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Tom Conahan, Assistant Director; Cathy Hamann; Amy Leone; Dan Ries; Rebecca Rust Williamson; and Jennifer Whitworth made key contributions to this report.