Centers of Excellence: DOD and VA Need Better Documentation of Oversight Procedures
Released: 2015-12-02T00:00:00

Letter

Background

The Assistant Secretary of Defense for Health Affairs leads the MHS and is ultimately responsible for the Defense COEs. In 2011, DOD leadership delegated responsibility for designating and overseeing Defense COEs to the Oversight Board. The Oversight Board includes members appointed by the Surgeons General of the Army, the Navy, and the Air Force; VA; and other components within DOD, as shown in table 1.

The Oversight Board’s charter, which was signed by the Assistant Secretary of Defense for Health Affairs, delegates oversight of Defense COEs to the Oversight Board. The charter lists the Oversight Board’s responsibilities and activities, and the quarterly meeting minutes provide documentation of the Oversight Board’s activities, procedures, and decisions. For example, the Oversight Board’s charter requires the Board to conduct periodic reviews of Defense COEs’ performance and to review and recommend applicants for Defense COE designation. The charter was originally signed in September 2011 by the Assistant Secretary of Defense for Health Affairs, and updated and signed again in May 2015. The updated charter clarified responsibilities for the Oversight Board related to overseeing COEs, such as validating that a Defense COE is meeting objectives for which it was established and that the return from its work merits continued investment.

For VA, VHA’s Under Secretary for Health is ultimately responsible for VHA COEs. Three program offices within VHA—the Office of Patient Care Services, the Office of Research and Development, and the Office of Academic Affiliations—have COEs. These three program offices delegate responsibility for their COEs to service offices within their organizational structures. The Office of Patient Care Services has three service offices with COEs—Mental Health Services, Specialty Care Services, and Geriatric and Extended Care Services. The Office of Research and Development has one service office with COEs— Rehabilitation Research and Development Service—and the Office of Academic Affiliations has two service offices with COEs, referred to as coordinating centers—Primary Care Education and Patient-Centered Specialty Care Education. (See fig. 1).

DOD Has Established Criteria for Designating a COE, While VHA Has Not

DOD officials established criteria that COEs must meet to be designated a Defense COE and a uniform process for applicants. VHA’s service offices use a peer review process to designate COEs. However, unlike DOD, VHA has not established criteria for an entity to be designated as a COE.

DOD Has Established Criteria for Defense COEs and Has a Uniform Process for Approving COE Applicants

Criteria for designating entities as COEs. In 2011, following our study of one of DOD’s statutorily mandated COEs, DOD leadership determined that it needed to conduct a review of all existing COEs and develop a definition that would be used as criteria for designating entities as Defense COEs. MHS leadership developed a definition for Defense COEs, and the COE Oversight Board members refined and approved the criteria contained in this definition. Only entities that meet the criteria in the Oversight Board-approved definition can be given the designation of a Defense COE.

The definition approved by the Oversight Board states that Defense COEs will focus on an associated group of clinical conditions and achieve improvement in outcomes through clinical, educational, and research activities. The criteria require Defense COEs to provide the entire clinical spectrum of care for a patient—from the prevention of diseases and treatment of clinical conditions through rehabilitation and transition to civilian life; for example, by developing clinical practice guidelines and educational materials and identifying research priorities and strategies for improving access to care. In addition, the Oversight Board developed other criteria that entities applying for Defense COE designation have to meet, such as clearly defining their mission, developing metrics to quantitatively assess their progress in meeting their mission, and determining whether the research they plan to conduct is needed because of existing research gaps.

The Oversight Board acting chairman said the board developed criteria for a Defense COE because it is important for the board, as well as MHS leadership, to apply consistent criteria when designating entities as Defense COEs. The acting chairman said not having clear and consistent criteria could make it easier for entities to self-identify as a Defense COE without meeting rigorous requirements. In addition, the criteria may facilitate coordination among COEs to meet the agency’s intended objectives for them—which is to improve the health of servicemembers, veterans, their families, and ultimately the military readiness of servicemembers.

Approval process for Defense COEs. The Oversight Board established a uniform process that requires applicants to present consistent information when applying for Defense COE designation. Applicants are required to describe why the designation as a Defense COE is important to them and how they meet the criteria of a Defense COE. The Oversight Board, after considering the applications from potential COEs, subsequently determines whether a Defense COE applicant passes the preliminary review. Applicants that do not pass the preliminary review are instructed to provide additional information or clarify the information presented for reconsideration. There is no limit on the number of times they can resubmit their application. Those that pass this preliminary review are instructed to develop a concept of operations (CONOPS) and a briefing for the Oversight Board. A CONOPS is a document that is designed to give an overall picture of the operation of the proposed Defense COE, explaining what the applicant intends to accomplish and how it will be done using available resources. The CONOPS includes a description of the value that the applicant brings to the MHS and a brief description of MHS needs and gaps that the applicant will address. The Oversight Board developed a CONOPS template to ensure that during required briefings information is consistently presented to the Board by applicants seeking Defense COE status. The applicant’s briefing to the Oversight Board is intended to provide an overview of the mission and goals of the applicant COE, including how the applicant meets the Defense COE criteria, as established by the Oversight Board.

After reviewing documentation from applicants, Oversight Board members make recommendations about Defense COE designation to the Assistant Secretary of Defense for Health Affairs, who makes the final decision. According to DOD officials, in 2012, after the Oversight Board reviewed the briefings and CONOPS for the four statutorily mandated COEs and made its recommendations, the Assistant Secretary of Defense for Health Affairs designated these applicants as Defense COEs. Subsequently, the Oversight Board reviewed and approved three other applicants as Defense COEs, according to these officials. The seven Defense COEs are listed in table 2 along with the origin of the Defense COE—that is, whether the Defense COE was statutorily mandated or departmentally designated.

VHA Has Not Developed Criteria for a COE, but Service Offices Use a Peer Review Process to Designate COEs

No criteria for designating entities as COEs. VHA has not developed consistent criteria for designating an entity as a COE. VHA officials told us they believe the term COE is “a term of art” and does not lend itself to standard and consistent criteria. Furthermore, officials said they never considered a need for these criteria.

Standards for Internal Control in the Federal Government provide that management should have a control environment that provides management’s framework for planning, directing, and controlling operations to achieve agency objectives, such as VHA’s objectives for how COEs are to operate and what COEs are supposed to achieve. A good internal control environment requires that the agency’s organizational structure clearly define key areas of authority and responsibility for operating activities. The organizational structure encompasses the operational processes needed to achieve management’s objectives. Without VHA developing criteria to establish, execute, control, and assess COEs, VHA management risks not meeting its objectives for COEs.

The lack of standard and consistent criteria for designating COEs hinders VHA’s ability to carry out the following functions.

VHA cannot provide both a basis for determining whether COEs are meeting the agency’s intended objectives for COEs and a coordinated direction for its COEs. These objectives include meeting the needs of veterans and their families, conducting pertinent research, and promoting innovative approaches to care delivery by VHA clinicians, according to VHA officials.

VHA officials might not be able to determine the precise number of COEs within the agency as a basis for planning, directing, and controlling operations to achieve agency objectives. VHA officials reported to us that the agency has 70 COEs, with the largest number located within VHA’s Office of Patient Care Services. The Office of Patient Care Services reported that it has 49 COEs—39 statutorily mandated and 10 VHA designated. The Rehabilitation Research and Development Service, under VHA’s Office of Research and Development, reported it has 13 VHA-designated COEs, and VHA’s Office of Academic Affiliations reported 8 VHA-designated COEs. However, VHA’s Office of Patient Care Services Chief of Financial Operations could not confirm this number, telling us he could not provide us with a definite number of COEs because there were no criteria for him to use to identify entities designated as COEs. Other VHA officials have also had difficulty identifying the universe of VHA’s COEs. For example, VHA officials initially omitted listing the 20 Geriatric Research Education Clinical Centers (GRECC) as VHA COEs until we told these officials that GRECC officials told us they were designated as VHA COEs.

In addition, confusion exists within VHA about statutorily mandated COEs because of the lack of criteria. For the COEs that VHA officials said were statutorily mandated, the statutory language often uses the term “center.” VHA decided to designate some of these centers as COEs, even though the statutory language was the same or similar for many centers that VHA did not designate as COEs. For example, the National Center for Preventive Health has statutory language similar to the language establishing the National Center for Post-Traumatic Stress Disorder (PTSD); however, the National Center for PTSD is considered a VHA COE while the National Center for Preventive Health is not. Officials from VHA and VA’s Office of General Counsel were unable to explain why some centers listed in statutory language are designated as COEs, while other centers with similar language are not designated as COEs. In addition, these officials could not provide the criteria that were used to designate these centers as COEs.

Process for designating entities as VHA COEs. VHA service offices use a peer review process to designate entities as VHA COEs. In general, a peer review process is often used by government agencies to determine the merit of proposals submitted by researchers applying for grants or some type of funding.

VHA service offices typically solicit applications or proposals from entities interested in being designated a COE, and interested entities complete an application or submit a proposal. Each application or proposal for COE designation may differ depending on the condition, disease, or specific health-related area being studied. For example, applications or proposals for a mental health COE designation may require applicants to address how their research will focus on bipolar disorder, borderline personality disorder, or schizophrenia, while applications or proposals for an educational COE may require applicants to address how they will develop and test innovative approaches for curricula related to patient-centered care or study new approaches and models of collaboration among health care professionals.

Within each of the six service offices, applications or proposals are reviewed by a panel of subject matter experts who prioritize the applications based on the strengths and weaknesses of the proposals or on the proposals with the highest merit rating. The service office peer review panel may be made up of experts from VHA entities or from entities external to VHA, according to VHA officials. Generally, the experts review information such as the focus of the planned research and available staffing and funding. Once the experts identify the best applicants, they forward this information to the appropriate service office officials. If the service office staff agrees with the list of best applicants, some service offices forward this list to the Under Secretary for Health, who ultimately makes the final decision with respect to designating an entity as a VHA COE, while other service offices forward the list to their program director, who makes the final decision.

While all six service offices use a peer review process to review, approve, and designate entities as COEs, the processes may differ in several respects. First, the content of submitted applications or proposals may differ among service offices. This is due, in part, to the lack of consistent and standard criteria within VHA that applicants must meet to be designated a VHA COE, such as how the applicant will meet the needs of veterans and their families and ensure that pertinent research is conducted to meet these needs—VHA’s intended objectives for its COEs, according to VHA officials. Second, the types and levels of review within service offices vary. For example, to help prioritize the best applicants, two VHA service offices developed a numerical scoring system to rate each application based on scientific and technical merit, typically based on the requirements contained in the solicitation for applications or proposals. For instance, if the solicitation for applications or proposals requires an evaluation plan that contains specific evaluation criteria, such as proposed outcome measurements, reporting methodology, expected findings, and potential implications for VHA and the community, applicants can be awarded up to 10 points for including these items. Other service offices do not include a scoring system as part of their approval process. Third, four of VHA’s six service offices conduct a site visit to the highest rated applicants’ facilities as part of their approval process; the other two do not, according to VHA officials. An official from one service office told us the staff members that conduct the visits have seen many potential COEs, and such on-site inspections can help to determine the applicant’s potential viability as a COE.

Both DOD and VHA Lack Written Procedures for Documenting Activities to Oversee COEs

Defense COE Oversight Board Does Not Have Written Procedures for Documenting Oversight Activities

Our review of the Oversight Board’s charter found that it does not contain procedures for how oversight of Defense COEs will be documented. Specifically, the Oversight Board charter does not explain (1) how the Oversight Board will provide and document its feedback to Defense COEs; (2) how the COEs will respond, if needed, to this feedback; and (3) how the Oversight Board will determine and document that the COEs’ actions resolved any identified problems. The acting chairman of the Oversight Board told us the board’s charter gives the board its authority to conduct oversight of Defense COEs, and if these types of procedures are needed, the Oversight Board’s charter and meeting minutes will serve this purpose.

The Oversight Board’s acting chairman said the board’s minutes document its activities and decisions, including the procedures followed when conducting COE oversight reviews and any problems identified. However, our review of the Oversight Board’s minutes, from its inception in 2011 to April 2015, shows that the minutes did not indicate the procedures followed when conducting COE reviews and did not explain how the Oversight Board documented and resolved identified problems.

The Standards for Internal Control in the Federal Government state that transactions and events should be promptly documented to maintain their relevance and value to management in controlling their operations and helping make decisions. Further, the standards state that significant events, such as in this instance the identification of problems during oversight and the actions taken to correct these problems, need to be clearly documented, and the documentation should be readily available for examination. Federal internal control standards also state that significant events should appear in management directives, policies, or operating manuals to help ensure management’s directives are carried out as intended. Once established, federal internal control standards state that management should monitor and assess over time the quality of performance, including monitoring the policies and procedures to ensure that the findings of reviews are promptly resolved.

Oversight Board officials told us that feedback to Defense COEs from oversight reviews conducted by the board is typically provided verbally and has not been documented. Therefore, documentation of feedback, both positive and negative, is not always available. Officials said that negative feedback may be documented in the Oversight Board’s minutes; however, if the Oversight Board identifies problems with the Defense COEs, the board’s charter and meeting minutes do not require that the Defense COEs provide written corrective action plans. As a result, there will not be a record of the corrective action taken by a Defense COE and whether the action resolved the problem identified by the Oversight Board. Absent specific procedures for how oversight should be conducted and how findings and corrective actions should be documented, DOD leadership lacks assurance that the Oversight Board has identified all problems and has taken appropriate action to determine that the problems have been resolved.

Most VHA Service Offices Lack Written Procedures for Documenting Oversight of COEs

Only one of six VHA service offices has written procedures for documenting the oversight of its COEs, including providing written critiques of findings from service office reviews to its COEs and requiring corrective actions from COEs when needed. The other five service offices do not have written procedures for documenting oversight activities. While most service offices do not have written procedures that require them to document their COE oversight, several currently provide written feedback to their COEs on the results of the service offices’ reviews. Specifically, three of the five service office directors provide written feedback to their COEs on the findings from the service office reviews. However, only one of these three service office directors requests that his COEs provide written corrective action plans.

Officials from three service offices told us they believe the process they currently have works fine because they provide written feedback to the COEs on the results of their reviews. However, if these directors leave the service offices, another director might not request written documentation of the oversight that is conducted because the documentation procedures are not written. The Standards for Internal Control in the Federal Government state that transactions and events should be promptly documented to maintain their relevance and value to management in controlling their operations and helping make decisions. Further, the standards state that significant events, such as in this instance the identification of problems during oversight and the actions taken to correct these problems, need to be clearly documented, and the documentation should be readily available for examination. Federal internal control standards also state that significant events should appear in management directives, policies, or operating manuals to help ensure management’s directives are carried out as intended. Once established, federal internal control standards state that management should monitor and assess over time the quality of performance, including monitoring the policies and procedures to ensure that the findings of reviews are promptly resolved. Without written procedures for documenting oversight activities, VHA leadership lacks assurance that its service offices are identifying and correcting all problems. Leadership also does not have evidence that the service offices conducted a review and do not have documentation of past and present problems to identify potential patterns and take action quickly to minimize the effects of these problems.

Conclusions

Unlike DOD, VHA has not developed standard criteria that entities must meet in order to be designated a VHA COE. Without defined criteria, VHA lacks reasonable assurance that its COEs are meeting the agency’s intended objectives for COEs, such as meeting the needs of veterans and their families throughout VA’s health care system and operating with coordinated direction.

By not having written procedures that outline how the agencies will document the activities through which they monitor and oversee the performance of COEs, both DOD and VHA lack assurance that oversight activities are performed consistently over time as intended. Written procedures would better ensure a common understanding of oversight activities among staff and enhance clear communication, especially as normal turnover occurs among the staff responsible for monitoring and providing feedback to COEs. Having systematic procedures for documenting oversight activities is necessary to better ensure that the agencies’ COEs are accountable for accomplishing the agencies’ objectives for them, such as meeting the health care needs of servicemembers and veterans.

Recommendations for Executive Action

To help ensure that COEs are meeting VHA’s intended objectives for them, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to establish clear, consistent standard criteria that entities must meet to receive COE designation, and require all existing VHA COEs, as well as new applicants for COE status, to meet these criteria.

To improve documentation of the activities DOD undertakes to oversee the Defense COEs, we recommend that the Secretary of Defense direct the Assistant Secretary of Defense for Health Affairs to require the MHS Defense COE Oversight Board to develop written procedures on how to document oversight activities of Defense COEs, including requirements for documenting feedback, both positive and negative, and documenting the resolution of identified problems.

To help improve VHA’s oversight of its COEs, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to require VHA service offices to develop written procedures on how to document their oversight activities of COEs, including requirements for documenting feedback, both positive and negative, and documenting the resolution of identified problems.

Agency Comments

VA provided written comments on a draft of this report, as well as an action plan for implementing our recommendations. We have reprinted VA’s comments and action plan in appendix III. In its comments, VA generally agreed with our conclusions and concurred with our recommendations. VA stated that a team of VHA subject matter experts will develop standards to be used in designating COEs and overseeing their performance.

DOD also provided written comments on a draft of this report, which we have reprinted in appendix IV. In its comments, DOD concurred with our findings and recommendation and explained how it intends to implement the recommendation. DOD also provided technical comments, which we have incorporated in the report as appropriate.

We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretary of Veterans Affairs; and other interested parties. We will also make copies available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: List of Veterans Health Administration Centers of Excellence and Their Focus

The Veterans Health Administration (VHA) within the Department of Veterans Affairs has three program offices—the Office of Patient Care Services, the Office of Research and Development, and the Office of Academic Affiliations—that have centers of excellence (COE). These three program offices delegate responsibility for their COEs to six service offices within their organizational structure. The Office of Patient Care Services has three service offices with responsibility for VHA COEs— Mental Health Services, Specialty Care Services, and Geriatric and Extended Care Services. The Office of Research and Development has one service office with COEs—Rehabilitation Research and Development Service—and the Office of Academic Affiliations has two service offices with COEs, referred to as coordinating centers—Primary Care Education and Patient-Centered Specialty Care Education.

VHA’s Office of Patient Care Services

VHA’s Office of Patient Care Services has all 39 of the COEs that were statutorily mandated, as well as 10 COEs that were departmentally designated. This office has three service offices that are responsible for these COEs: Mental Health Services, Specialty Care Services, and Geriatrics and Extended Care Services. Table 3 lists each COE in the Office of Patient Care Services and groups the COEs by their specific service office, as well as indicating their location. The table also indicates the origin of the COE—whether the COE was statutorily mandated or departmentally designated—and provides a brief description of the COE’s research, clinical, and/or educational focus, as provided by VHA.

VHA’s Office of Research and Development

VHA’s Office of Research and Development has 13 COEs that were departmentally designated, according to VHA officials. This office has one service office—Rehabilitation Research and Development—that is responsible for these COEs. Table 4 lists each COE in this service office and provides the location of the COE, as well as a brief description of the COE’s research, clinical, and/or educational focus, as provided by VHA.

VHA’s Office of Academic Affiliations

VHA’s Office of Academic Affiliations has eight COEs that were departmentally designated, according to VHA officials. This office has two service offices, referred to as coordinating centers—Primary Care Education and Patient-Centered Specialty Care Education—that are responsible for COEs. Primary Care Education has five COEs and Patient-Centered Specialty Care Education has three COEs. Table 5 lists each COE by service office and gives the location of the COE and a brief description of the research, clinical, and/or educational focus of the respective COEs.

Appendix II: Collaboration Efforts of Centers of Excellence in the Department of Defense and Department of Veterans Affairs

To describe the collaboration efforts of the Defense Centers of Excellence (COE) in the Department of Defense (DOD) and the Veterans Health Administration (VHA) COEs in the Department of Veterans Affairs (VA), we sent a Web-based, structured questionnaire to COE directors, identified by DOD and VHA officials, to obtain information about how their COEs collaborate. We sent the questionnaire to the COE directors between December 2014 and January 2015. The questionnaire asked COE directors to describe the extent to which Defense and VHA COE staff collaborate internally—with other staff from within their agencies, and externally—with staff from other federal agencies and academic organizations. The questionnaire also asked the COE directors if they use certain tools, such as written agreements, staff participation in committees, working groups, councils, or task forces; or other tools or mechanisms to coordinate or collaborate. All 7 Defense COE directors responded to the questionnaire, and 60 of VHA’s 70 COE directors, or 86 percent, responded to the collaboration section of the questionnaire. Tables 6 through 11 provide information about Defense and VHA COE collaboration efforts.

Defense COEs

Defense COEs report using written agreements or other tools to collaborate. Table 6 shows the Defense COEs and their reported collaboration activities.

VHA’s Office of Patient Care Services

VHA’s Office of Patient Care Services has three service offices that have established 49 COEs, with 39 of them statutorily mandated and 10 departmentally designated, according to VHA officials. The service offices are the Mental Health Services, the Specialty Care Services, and the Geriatric and Extended Care Services. The COEs that responded to the collaboration section of the questionnaire report using written agreements or other tools to collaborate. Tables 7, 8 and 9 provide information on the collaboration activities of the three service offices within the Office of Patient Care Services that have COEs.

VHA’s Office of Research and Development

VHA’s Office of Research and Development has one service office that has COEs—the Office of Rehabilitation Research and Development Service. This service office reports they have 13 COEs, all departmentally designated, that focus on a selected area of research relevant to veterans with disabilities. The COEs report using written agreements or other tools to collaborate. Table 10 indicates the collaboration activities of the COEs.

VHA’s Office of Academic Affiliations

VHA’s Office of Academic Affiliations reports that it has eight COEs and all were departmentally designated. This office has two service offices, referred to as coordinating centers: Primary Care Education and Patient- Centered Specialty Care Education. Primary Care Education has five COEs and Patient-Centered Specialty Care Education has three. The Primary Care Education and Patient-Centered Specialty Care Education COEs that responded to the collaboration section of our questionnaire indicated that they use written agreements or other tools to collaborate. Table 11 shows the collaboration activities of the COEs that responded to our questionnaire.

Appendix III: Comments from the Department of Veterans Affairs

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact:

Staff Acknowledgments

In addition to the contact named above, Marcia A. Mann, Assistant Director; Mary Ann Curran Dozier; Martha Fisher; Carolyn Fitzgerald; Carolina Morgan; and Jacquelyn Hamilton made key contributions to this report.