VA and DOD Health Care: First Federal Health Care Center Established, but Implementation Concerns Need to Be Addressed
Released: 2011-07-19T00:00:00

Letter

Background

VA and DOD’s history of sharing resources to provide integrated health care services to their beneficiaries in and around North Chicago has occurred in three phases. The third phase culminated in the establishment of the FHCC, which was formalized by an Executive Agreement signed by the Secretaries of VA and DOD in April 2010. The FHCC is unique among other VA and DOD sharing relationships in its level of collaboration, governance structure, and financial model.

History of VA and DOD Integration of Health Services in North Chicago

The history of VA and DOD integrating health services in North Chicago can be described in three distinct phases. (See fig. 1.) These phases cover the time period from 2003 to 2011 and include the official establishment of the FHCC in October 2010.  Phase I began in 2003 when VA and DOD began sharing health care resources between NCVAMC and NHCGL. NHCGL, then known as Naval Hospital Great Lakes, transferred its inpatient mental health services to NCVAMC. In the same year, VA and DOD formed a working group to address issues related to sharing resources between the two sites. In 2004, the NHCGL blood donor processing center was transferred to NCVAMC.  Phase II began in 2005 with the $13-million renovation and modernization of the NCVAMC, including its operating rooms and an expansion of the emergency department. In 2006, NHCGL’s inpatient medical, surgical, pediatric, and intensive care units, operating rooms, and emergency department were transferred to NCVAMC. With the transfer of inpatient services, the naval hospital became a naval health clinic, since the facility no longer provided inpatient services.  Phase III began in 2007 when construction began on new parking areas, followed in 2008 by the groundbreaking for the construction of a new ambulatory care center. The FHCC Advisory Board was established in 2009 to help provide guidance for the integration and future operation of the facility, which was authorized in the NDAA for Fiscal Year 2010. In October 2010, the former NCVAMC and NHCGL facilities merged to become the FHCC, following completion of a $130-million DOD-funded construction project. The FHCC consists of all the services, buildings, and locations formerly operated by either NCVAMC or NHCGL including the new 201,000-square-foot ambulatory care center and its parking lot and garage, a 45,000- square-foot renovation of the NCVAMC, and various outpatient and recruit clinics formerly operated by either NCVAMC or NHCGL. (See fig. 2 for photographs of newly constructed or renovated areas of the FHCC.) The ambulatory care center, which is physically connected to the NCVAMC, houses outpatient services including pediatrics, women’s health, and mental health. In addition, it has on-site laboratory, radiology, and pharmacy services, enabling patients to access these ancillary services in the same location as their outpatient services. Phase III continued into 2011 with the move into the new ambulatory care center and the delivery of patient services there.

In April 2010, the Secretaries of VA and DOD signed the Executive Agreement that established the FHCC. The Executive Agreement defines the relationship between VA and DOD for establishing and operating the FHCC, in accordance with the NDAA for Fiscal Year 2010, and contains provisions in 12 integration areas regarding specific aspects of FHCC operations. These 12 areas are: (1) governance structure; (2) access to health care at the FHCC; (3) research; (4) contracting; (5) information technology (IT); (6) fiscal authority; (7) workforce management and personnel; (8) quality assurance; (9) contingency planning; (10) integration benchmarks; (11) property (i.e., construction and physical plant management); and (12) reporting requirements.

Within the 12 areas are provisions describing how the FHCC should be operated or what VA and DOD will do as part of their efforts to jointly operate the facility. Some provisions relate to establishing and operating the FHCC and have designated deadlines, such as implementing IT strategies. Other provisions do not have specified deadlines or will not be met until a certain point in the integration, or are contingent on other conditions being met. For example, a provision in the reporting requirements integration area calls for a final report at the end of the 5-year demonstration in 2015. Since the report is due at the end of the 5-year demonstration period, this particular provision cannot yet be implemented.

The Executive Agreement also includes 15 integration benchmarks that VA and DOD plan to use to determine the integration’s success. Assessment of the integration benchmarks throughout the 5-year demonstration project will help inform whether the FHCC partnership should continue beyond the demonstration period authorized by the NDAA for Fiscal Year 2010.

Unique Features of the FHCC

The FHCC is unique among VA and DOD joint ventures in three key ways. First, the FHCC’s integration of the provision of care and operations represents the highest level of collaboration among the nine existing VA and DOD joint ventures. VA and DOD have periodically assessed their joint venture arrangements to determine the level of collaboration between partners, which they measure on a continuum from “separate” to “consolidated.” They use this continuum to assess program elements of a joint venture’s partnership such as governance, education and training, and research. VA and DOD officials reported to us that overall, the FHCC has more program elements, such as its clinical services and staffing, which fall on the “consolidated” end of the collaboration continuum than any of the other joint venture sites. Second, the FHCC operates under a single line of governance to manage medical and dental care, and has an integrated workforce of approximately 3,000 civilian and active duty military employees from both VA and DOD—a feature that is unique to the FHCC. Although the FHCC’s leadership and workforce are integrated, they also remain directly accountable to both VA and DOD through the Joint Executive Council and the Health Executive Council. VA and DOD officials told us that none of the other joint venture sites has an integrated governance structure and instead maintain separate VA and DOD lines of authority. The third way in which the FHCC is unique among VA and DOD joint ventures is its financial model. The FHCC has a single funding source to which VA and DOD will contribute, unlike the other joint venture sites which have separate VA and DOD funding sources. The NDAA for Fiscal Year 2010 established the Joint DOD-VA Medical Facility Demonstration Fund (Joint Fund) as the funding mechanism for the FHCC, with VA and DOD both making transfers to the Joint Fund from their respective appropriations.

FHCC Officials Have Made Progress Implementing VA and DOD’s Executive Agreement, but Challenges May Impact Further Implementation Progress

FHCC officials (including both VA and DOD officials) have made progress implementing the provisions of the 12 Executive Agreement integration areas. Four of the 12 integration areas have been fully implemented, 7 are in progress and proceeding according to plan, and 1 area, IT, has been delayed and continues to present challenges. In addition, the FHCC’s lack of a military treatment facility (MTF) designation has presented challenges for FHCC operations and health care providers.

FHCC Officials Have Implemented Four Executive Agreement Integration Areas and Are Progressing on Seven Others, but IT Implementation Has Been Delayed

FHCC officials have addressed all provisions for 4 of the 12 Executive Agreement integration areas, are progressing in their implementation of 7 other areas, and have experienced delays in the implementation of 1 area, IT. (See table 1.)

Officials have completed implementation of the governance structure, access to health care at the FHCC, research, and contracting integration areas.

Governance structure. The Executive Agreement defined the structure of the FHCC’s governance structure and leadership, with a VA official serving as the director and a naval captain serving as the deputy director. (See fig. 3.) As of October 1, 2010, the former director of NCVAMC became the director of the FHCC, and a command change brought in a new naval captain to become deputy director, taking over command from the former NHCGL commanding officer.

In addition, the advisory bodies described in the Executive Agreement are in place. The FHCC has an Advisory Board—co-chaired by and comprised of senior officials from VA and DOD—that monitors the FHCC’s performance and advises on strategic direction, mission, vision, and policy. The Advisory Board also provides input into the performance evaluations of FHCC leadership and serves as a communication link between VA and DOD executive leadership and the FHCC through the Joint Executive Council and the Health Executive Council. Also, a Stakeholders Advisory Council—comprised of members from various regional and local organizations representing FHCC interests—provides feedback on how well the FHCC is meeting customers’ needs and whether the FHCC is meeting VA and DOD missions. VA and DOD officials also designed the major operational components of the FHCC to have shared VA and DOD leadership. The six clinical and administrative divisions that report to the director and deputy director are led by an associate director and an assistant director, one from VA and the other from the Navy.

Access to health care at the FHCC. To address access to health care at the FHCC for veterans and DOD beneficiaries, the NDAA for Fiscal Year 2010 and the Executive Agreement established a patient priority system that is unique to the FHCC for use in the event of resource constraints. FHCC’s patient priority system is based on the priority list for TRICARE, DOD’s program to provide health care to its beneficiaries, and incorporates VA beneficiaries. More specifically, the system prioritizes active duty servicemembers above veterans and other DOD beneficiaries as follows: 1. members of the Armed Forces on active duty; 2. veterans and non veteran VA beneficiaries, and TRICARE Prime- enrolled active duty dependents; 3. TRICARE Prime enrolled retirees, their dependents and survivors; 4. TRICARE Standard active duty dependents; and 5. TRICARE Standard retirees, their dependents, and survivors, including TRICARE for Life beneficiaries.

Officials told us that they do not anticipate needing to activate the patient priority system because they are currently meeting the needs of FHCC beneficiaries—health care providers at the FHCC currently serve all patients based on medical need. Officials also told us that their monitoring of Navy recruit medical readiness ensures they are able to maintain the “pipeline to the fleet” of enlisted sailors.

Research. The Executive Agreement stated that the FHCC would comply with VA policy for research efforts, but provided that when DOD researchers or patients are involved in a study, the Navy’s rules on protection of human subjects would apply in addition to VA’s. In addition to implementing this provision, FHCC officials told us they decided to integrate the research program at the FHCC. Since a majority of the research conducted is VA research, it was easily incorporated into the broader FHCC integration efforts. The FHCC has an Institutional Review Board—a body responsible for reviewing and approving research protocols involving human subjects—located at a hospital affiliated with the FHCC that provides research management and operational oversight to the FHCC, the Edward Hines, Jr. VA Hospital in Hines, Illinois. Also, in accordance with the Executive Agreement, FHCC officials told us that a DOD Institutional Review Board in San Diego, California, may also be involved for research involving DOD researchers or active duty servicemembers at the FHCC.

Contracting. The Executive Agreement stated that VA would be responsible for providing contracting support at the FHCC. Similar to the FHCC’s research efforts, officials chose to integrate the contracting function as part of the broader integration. Five former Navy civilian employees who were at NHCGL prior to the integration were converted to VA civilian employees to help support the integrated contracting and purchasing functions.

Officials’ efforts have progressed as planned to implement provisions for the fiscal authority, workforce management and personnel, quality assurance, contingency planning, integration benchmarks, property, and reporting requirements areas.

Fiscal authority. The FHCC fiscal authority integration area included the development of an integrated budgeting and financial reconciliation process. For fiscal years 2011 through 2013, the FHCC plans to use historical financial data to budget and determine the amount each department will transfer to the Joint Fund and expects to manually conduct the year-end reconciliation process. Officials told us that by fiscal year 2014, the FHCC intends to have an automated year-end financial reconciliation process. However, as of April 1, 2011, the integration area on fiscal authority had not been fully implemented because appropriations had not been made available for the Joint Fund. The NDAA for Fiscal Year 2010 established the Joint Fund as the FHCC’s funding source, but FHCC officials could not use it until funds had been authorized and appropriated for VA and DOD to transfer into the Joint Fund, which occurred in April 2011. Until that time, the FHCC was funded by an alternative funding mechanism established by the Executive Agreement for use in the event that Congress did not authorize and appropriate funds to be transferred to the Joint Fund. As of April 2011, FHCC officials planned to cease use of the alternative funding mechanism and begin use of the Joint Fund at the start of the next quarter on July 1, 2011.

The delay in availability of funds may result in a delay in addressing an NDAA for Fiscal Year 2010 requirement that is also one of the integration benchmarks—the annual independent audit of the Joint Fund, which is conducted at the end of the fiscal year. The audit will evaluate the adequacy of VA’s and DOD’s proportional contributions to the Joint Fund. In addition, the implementation of the automated financial reconciliation process is contingent on a related IT capability, which does not yet have an estimated completion date. Together, these delays may impact the FHCC’s ability to address one of the measures of its integration’s success.

Workforce management and personnel. In the workforce management and personnel integration area, the NDAA for Fiscal Year 2010 authorized a transfer from DOD to VA of the positions and personnel necessary to operate the FHCC. The Executive Agreement identified 533 DOD civilian positions that were eligible for transfer to VA, and FHCC officials told us that VA made offers of employment to the individuals in those positions. In total, 469 DOD civilian personnel were transferred to VA as of October 10, 2010—the deadline established in the Executive Agreement. The 533 converted civilian positions, along with 724 active duty positions, 1577 VA civilian positions, 249 VA and DOD contract positions, and 18 new housekeeping positions, comprise the approximately 3,000 positions that initially staffed the FHCC. FHCC officials are in the process of resigning affiliation agreements with health care facilities and training institutions and plan to address another provision regarding the development of criteria and assessment methods to measure staff experiences with the integration at a later date. Officials have also integrated their staff training through an integrated education department.

Quality assurance. The Executive Agreement stated that the FHCC would have one integrated quality assurance plan and would maintain accreditation by the external accrediting bodies required by either VA or DOD. It also outlined the FHCC’s credentialing and privileging process for health care professionals. FHCC officials have an integrated quality assurance plan for the facility in place, as well as policies addressing credentialing and privileging of providers and the role of independent duty corpsmen at the FHCC. While some bodies, such as the VA Office of the Inspector General, have conducted reviews of the FHCC since it was established, other accreditation and certification reviews, such as that of The Joint Commission, are pending and the first reviews for the FHCC will occur on the schedules that were in place for the NHCGL and NCVAMC prior to the integration.

One component of quality assurance is the maintenance of clinical skills for the FHCC’s Navy health care providers. Officials told us that one of the benefits of the integration is that dental school graduates obtaining advanced education in the Navy can see veteran patients while completing their residencies and have opportunities to be exposed to different dental conditions than those normally seen in the generally younger and healthier recruit population. Some of these dentists will be placed on ships, where they are often the only on-site dentist. FHCC officials described a similar benefit for health care professionals providing inpatient care.

Contingency planning. The Executive Agreement included contingency planning provisions regarding the establishment of certain FHCC emergency management positions, and stated which antiterrorism and other security guidelines would inform the establishment of the FHCC’s security plans. In addition, the FHCC must maintain training standards for staff that meet the joint VA/DOD programs in this integration area. The FHCC has the necessary emergency management personnel, training standards, and programs in place; however, officials are in the process of finalizing an agreement to outline the relationship between VA police and DOD security personnel.

Integration benchmarks. The Executive Agreement established 15 integration benchmarks to define the degree of the integration’s success. VA and DOD officials at the regional and headquarters levels and FHCC officials worked together to develop these benchmarks that cover such topics as patient and staff satisfaction, clinical and administrative functions, and external evaluation. The integration benchmarks are being used by the FHCC to assess provision of care and operations and are discussed in more detail later in this report.

Property. The Executive Agreement and the NDAA for Fiscal Year 2010 describe the terms of the transfer of property ownership that may occur at the end of the 5-year demonstration period from DOD to VA. A determination to transfer ownership may occur upon the earlier of (1) completion of the integration benchmarks or (2) 5 years from the date the Executive Agreement was executed. If it is determined that the FHCC should not continue to be an integrated facility, DOD will retain ownership of the ambulatory care center and associated structures that were built with DOD funds.

Reporting requirements. There are several reporting requirements described in the Executive Agreement that were established by the NDAA for Fiscal Year 2010, including submitting the Executive Agreement to the appropriate committees of Congress 1 week before its execution, and a final report from the Secretaries of VA and DOD that will be submitted 5 ½ years after the Executive Agreement was executed. The report is to describe and assess the performance of the FHCC, and to provide a recommendation as to whether the partnership should continue beyond the demonstration period. Congress will make the final determination as to whether to continue the partnership.

The Executive Agreement identified IT capabilities that VA and DOD were to have in place by the opening day of the FHCC, October 1, 2010, to facilitate interoperability between VA and DOD electronic health record systems, as well as other capabilities for financial management and outpatient appointments that are to be developed in the future. The three capabilities that were to be in place upon the FHCC’s opening were (1) medical single sign-on—which allows staff to use one screen to access both the VA and DOD electronic health record systems; (2) single patient registration—which allows staff to register patients in both systems simultaneously; and (3) orders portability for laboratory, radiology, pharmacy, and consults—which will allow VA’s and DOD’s electronic health record systems to exchange information for these medical orders. In addition, the Executive Agreement stated that all IT capabilities developed for the FHCC will be exportable to other VA/DOD joint ventures and medical sharing locations.

FHCC officials, working with VA and DOD officials, have implemented or are in the process of implementing the IT provisions of the Executive Agreement, including working with a strategic working group that supports implementation efforts, as well as defining procedures for the reporting of information security incidents. However, VA and DOD did not meet designated deadlines for the three capabilities that were to be in place upon opening and, as of May 2011, not all capabilities are fully implemented at the FHCC. Single sign-on and single patient registration were implemented on December 13, 2010. On March 3, 2011, FHCC officials began limited use of orders portability for laboratory and radiology. While full operational capability was expected on April 14, 2011, officials told us that both orders portability capabilities remained in limited use through April, with radiology expected to have full operational capability on June 1, 2011, and laboratory delayed until an undetermined date. FHCC officials decided to delay implementation of these capabilities in order to allow more time to correct problems, such as difficulty managing large numbers of automated laboratory test orders, and to train users on the system. Additionally, FHCC officials told us that implementation of the remaining orders portability capabilities (pharmacy and consults) are indefinitely delayed while decisions are made at the department level regarding development of these capabilities. FHCC officials have implemented an interim orders portability process for the pharmacy while VA and DOD continue to develop the automated orders portability capability. This interim process necessitated the hiring of five full-time pharmacists to conduct manual checks of the VA and DOD electronic health record systems to ensure that the FHCC is able to ensure patients’ safety by identifying possible interactions between drugs prescribed in the two separate systems.

The three IT capabilities were delayed in part because of a need for more on-site testing before use, as well as a lack of an integrated and comprehensive project plan from VA and DOD. During on-site testing, FHCC officials found that some requirements for pharmacy and radiology orders portability did not meet the FHCC’s needs. FHCC officials told us that an earlier round of system testing was performed in an off-site environment that did not effectively simulate the FHCC environment. In addition, we reported in February 2011 that, although VA and DOD performed various planning activities for the FHCC IT system, these activities generally were not completed in accordance with effective project planning practices including defining the scope, estimating the cost, and establishing a budget and schedule for the project. Additionally, we expressed concern that VA’s and DOD’s ineffective planning jeopardized their ability to fully provide the IT system capabilities the FHCC needs on a timely basis.

As a result of the need for more on-site testing and the ineffective project planning, VA and DOD have not yet fully provided clinicians at the FHCC with the IT capabilities the Executive Agreement identified as needed upon opening. Further, the IT issues caused a 2-week delay in the start of the move of clinical services into the new ambulatory care center. Additionally, since the needed capabilities have not yet been fully implemented, the departments are not in a position to export all the planned capabilities to other locations, as provided for in the Executive Agreement.

Lack of MTF Designation Has Presented Challenges for FHCC Operations and Health Care Providers

According to DOD policy, an MTF is a military treatment facility owned and operated by DOD that is established for the purpose of furnishing medical and/or dental care to eligible individuals. Among other things, designation as an MTF allows co-payments to be waived for services received by DOD beneficiaries. The former NHCGL was an MTF, and FHCC officials will continue to list it as such on DOD’s list of MTFs through the FHCC demonstration period even though the NHCGL no longer exists. The FHCC’s ambulatory care center, which is currently owned by DOD, is the only part of the FHCC that has an MTF designation. The NDAA for Fiscal Year 2010 provided that the facility, defined for purposes of the statute as the new Navy ambulatory care center, parking structure, and supporting structures and facilities, as well as related medical personal property and equipment, may be treated as an MTF for purposes of eligibility for DOD health care. However, DOD has concluded that it does not have the authority to designate the FHCC as an MTF since the FHCC generally, including those areas of the FHCC that provide inpatient services, is owned by VA and that its authority to consider the FHCC an MTF is limited to the purpose of confirming the categories of beneficiaries eligible for DOD health care.

FHCC officials, working with VA and DOD, have implemented or are in the process of implementing workarounds for three issues related to the lack of an MTF designation: 1. Certain DOD beneficiaries would have been responsible for co- payments for care received at the FHCC. DOD beneficiaries do not have to pay co-payments at MTFs, such as the former NHCGL. However, because the FHCC lacks the MTF designation, certain DOD beneficiaries would have had to pay co-payments for services received at the FHCC. This issue was temporarily resolved through a demonstration project to waive co-payments for DOD beneficiaries at the FHCC during the 5 years of the FHCC demonstration. 2. The FHCC was unable to continue DOD’s personal services contracts. FHCC officials told us that prior to the integration, the naval facilities used personal service contracts for temporary health care provider staffing needs, but DOD may only enter into personal services contracts to fulfill health care needs at MTFs and in other select circumstances. FHCC officials decided to convert the personal services contract positions that were needed at the FHCC into VA civilian employee positions. FHCC officials told us that personal services contracts are a preferred method for accommodating fluctuations in medical and dental workload resulting from changes in the number of Navy recruits on site at any given time, but they anticipate that the Navy’s plan to maintain more consistent recruit numbers throughout the year will reduce the need for temporary staff. 3. The FHCC’s lack of MTF designation resulted in uncertainty about the FHCC’s ability to use DOD’s contracted drug prices for prescription orders, including the extent to which DOD’s drug pricing information could be shared with VA’s pharmacy vendor. VA and DOD have contracts with pharmacy vendors to obtain drugs and with manufacturers to obtain favorable drug pricing for their beneficiaries, but the two departments have contracts with different vendors and have different pricing arrangements with manufacturers. As part of the broader integration efforts at the FHCC, VA and DOD signed an agreement to use VA’s pharmacy vendor for the FHCC while maintaining access to DOD’s contracted manufacturer prices for DOD beneficiaries treated at the FHCC. However, FHCC officials told us they were later denied access to manufacturer pricing arrangements because the FHCC was not an MTF. DOD officials told us that the TRICARE Management Activity’s Pharmacy Operations Department determined in March 2011 that the FHCC was entitled to use DOD- contracted prices with respect to DOD beneficiaries and issued a letter to manufacturers indicating that VA’s pharmacy vendor would use these prices. DOD officials told us that they plan to specifically include the FHCC in future manufacturer pricing arrangements, and in the meantime DOD will evaluate any objections received from manufacturers.

While workarounds have helped address certain MTF designation-related issues, another MTF designation-related issue continues to pose challenges for FHCC operations and health care providers. Navy Bureau of Medicine and Surgery policy regarding the deployment readiness of Navy and Marine Corps servicemembers requires the approval of a medical screener assigned to an MTF. The commanding officer of an MTF assigns specific providers at the facility (medical officers, physician assistants, nurse practitioners, or Independent Duty Corpsmen) the responsibility to conduct suitability and medical assignment screening. For example, for a Navy or Marine Corps servicemember to be approved for overseas duty, the servicemember must have a medical, dental, and educational suitability screening, and MTF medical and dental screeners must sign off on the form stating that the servicemember is suitable for that assignment. FHCC officials told us that while the forms are being signed, there is uncertainty as to whether providers continue to have the authority to sign the forms as MTF medical screeners because the FHCC is not an MTF, and they have not seen documentation that confirms whether provider sign-off authority has changed with the establishment of the FHCC. FHCC officials said this has created confusion among FHCC providers about how to interpret DOD policies regarding these documents.

VA and DOD Use Integration Benchmarks to Assess Provision of Care and Operations at the FHCC, but the Performance Reporting Plan May Not Yield Transparent, Meaningful, and Accurate Results

VA and DOD, through FHCC staff, are using the 15 integration benchmarks set forth in the Executive Agreement (and their corresponding performance measures) to assess the provision of care and operations at the FHCC. The plan is to report on these performance measures using a tool developed by FHCC staff—a scorecard that generates a monthly summary score. However, the summary score does not account for data collection variation, FHCC staff have not specified what target score(s) would indicate successful performance, and the scorecard initially contained an error, all of which raise concerns about the FHCC’s ability to report transparent, meaningful, and accurate performance results.

VA and DOD Are Assessing Provision of Care and Operations at the FHCC through 15 Integration Benchmarks

The 15 Executive Agreement integration benchmarks, chosen by VA and DOD, are intended to assess the provision of care and operations at the FHCC in three main areas of focus: patient and staff satisfaction, including benchmarks that measure patient and staff feedback; clinical and administrative functions, such as benchmarks aimed at assessing patient access to care and clinical productivity; and external evaluation, including our review among others. The benchmarks vary in several aspects including whether they were created specifically for the FHCC or whether they are compared to historical performance before the facilities were integrated, as well as the frequency of data collection for each individual benchmark and the specific performance measures each includes.

The 15 integration benchmarks vary by time frame of establishment. Most (9) of the 15 integration benchmarks were used by the former NCVAMC and NHCGL prior to the establishment of the FHCC, while the remaining benchmarks (6) were established specifically for the FHCC. (See table 2.) Of the benchmarks that pre-date the establishment of the FHCC, some have separate measurements for VA and DOD populations, such as patient satisfaction surveys, as was the case before the FHCC was established. FHCC officials said they have no short-term plans to integrate patient satisfaction, because separate measurements allow them to compare results from before and after the FHCC integration. In addition, the other benchmarks that pre-date the FHCC integration are measured for only VA or DOD populations. For example, DOD measures Navy servicemember medical readiness for duty, which was previously in place at the former NHCGL to help assess performance of DOD’s “mission critical” operational readiness goals. VA measures health profession trainee satisfaction, which had been measured at the former NCVAMC and helps to assess VA’s clinical and administrative performance. The 6 remaining benchmarks were created specifically for the FHCC, some of which were designed to measure aspects of FHCC integrated performance, such as whether the “information technology solution timeline is met and has no negative impact on patient safety.” Many of these benchmarks have no historical data to which performance can be compared.

The 15 integration benchmarks are comprised of 38 individual performance measures. Each of the integration benchmarks has corresponding performance measures, for a total of 38 individual performance measures for the 15 benchmarks. For most of the integration benchmarks, there are at least two individual performance measures. (See table 3.) FHCC staff have developed a Technical Manual to document their plan for the measurement, data collection, and reporting of these performance measures.

The 38 performance measures vary by frequency of data collection. FHCC staff collect data for the performance measures at different time intervals. (See fig. 4.) Depending on the individual measure, data generally are collected weekly, monthly, quarterly, annually, or every 2 to 3 years. Data for one performance measure is collected both semiannually and annually, and data may also be collected on a varied time frame.

In some cases, data for different performance measures within a single integration benchmark may be collected at different points in time. For example, data for the performance measures within the benchmark “satisfactory facility and clinical inspection, accreditation, and compliance outcomes from external oversight/groups” vary as to the times that the respective reviews are conducted; the Joint Commission, for example, conducts full reviews every 3 years. In addition, data collection for some measures has not yet begun. For example, the “validation of FHCC fiscal reconciliation model by annual independent audit” cannot yet be measured because audits have yet to be performed.

Reporting a Summary Score of Monthly Results May Not Yield Transparent, Meaningful, and Accurate Performance Information

FHCC staff have developed a reporting tool in the form of a scorecard that tracks and summarizes performance data for all 38 performance measures. The scorecard is designed to calculate scores for each of the performance measures as well as to generate a summary score every month. Each of the 38 performance measure scores is determined by multiplying: (1) a rating based on performance, (2) assigned weights based on the level of importance, and (3) a fixed multiplier to adjust the score to a scale of 100. These individual performance measure scores are then combined into a monthly summary score, also measured on a scale from 0 to 100. (See fig. 5.)

Specifically, each performance measure’s score is determined by the following: 1. Each performance measure is rated on a scale from 1 (lowest rating) to 5 (highest rating) based on how well the measure meets its target goal, according to definitions set in the FHCC Technical Manual. 2. The Advisory Board weighted each performance measure, assigning the greatest weight to measures they determined were most critical for meeting NDAA for Fiscal Year 2010 requirements. Specifically, those measures concerning DOD’s missions of servicemember readiness and VA’s mission of clinical and administrative performance received the greatest weights. The three weights used, from low to high, are 1 (important), 2 (essential), and 5 (critical). 3. In addition to tracking the trends for each of the performance measures, the scorecard is designed to calculate a summary score on a scale of 0 to 100 by adjusting each measure using a fixed multiplier. (See text box below for an example of the calculation of one performance measure score.)

Sample Performance Measure Score Calculation Using the example of the health profession trainee satisfaction performance measure, a score is determined using the following information: 1. FHCC staff assigned it a performance rating of 4 – health professionals are “somewhat satisfied” with training. (This is a baseline rating made before the FHCC was integrated on October 1, 2010.) 2. The performance measure has a weight of 1 (important). 3. All performance measures are adjusted to the 100 point scale using a fixed multiplier of 0.2128.

The rating and weight are multiplied together and adjusted to 100 for a score of 0.8512 (4 x 1 x 0.2128).

FHCC officials told us they designed the scorecard to calculate a user- friendly, single summary score on a monthly basis and report performance to the Advisory Board at their regular meetings, which are typically quarterly. However, we identified three areas of concern with the monthly summary score: The monthly summary score does not account for varied data collection time frames. Although the summary score is calculated monthly, data for all performance measures are not collected on a monthly basis. Specifically, FHCC staff told us they record no score when no new monthly data are available for a given performance measure, even when that measure was not expected to be collected on a monthly basis. In fact, there may be no single month where complete performance data are available to be factored into the summary score. For instance, the health profession trainee satisfaction performance measure, collected annually, would appear in the scorecard only in the month it was collected and have no data listed and no rating given for the remaining 11 months of the year. For more than half the performance measures, data are collected less frequently than monthly and would be similarly affected. In any given month, fluctuations in the summary score may be caused by varied data collection, and not changes in performance, which is not transparent in the scorecard methodology.

The summary score lacks a set target score(s) to indicate success. The summary score’s ability to provide a meaningful indication of success is unclear because neither FHCC staff nor the Advisory Board to whom the scores are reported has established any specific target score(s) to indicate that the FHCC has achieved success. FHCC officials told us that the goal of the scorecard is to calculate a summary score for all 38 performance measures on a scale of 0 to 100 to indicate the level of success of the integration, with a maximum (perfect) score of 100. While there are specific targets for each of the 38 performance measures, officials have not determined what score(s) will indicate overall success of the integration at the end of the 5-year demonstration. Without establishing a target summary score(s) to indicate successful FHCC integration, FHCC staff do not have the ability to gauge progress, thus diminishing the usefulness of calculating a summary score.

The calculation error raises concerns about accuracy in the scorecard methodology. Upon review of the FHCC’s final version of the scorecard, we discovered an error in the multiplier used to adjust the scores to a 100 point scale. With the addition or deletion of performance measures, the multiplier needs to be recalculated to ensure that the summary score retains a 100 point scale. We found that when the FHCC added performance measures, which now total 38, they had not adjusted the multiplier accordingly. This resulted in a scorecard that calculated a summary score with a possible total of 119 rather than 100, as the FHCC intended. Although the FHCC has fixed the error and in March 2011 presented a corrected scorecard to use going forward, the lack of initial awareness of the error raised concerns about the accuracy of the results reported using the scorecard.

Conclusions

The FHCC is a 5-year demonstration project that has the potential to be a model for future VA and DOD integration efforts. However, IT implementation delays, the lack of MTF designation, and concerns about the use of a summary score to report on FHCC performance may impact FHCC officials’ ability to provide the information necessary for Congress to determine whether to continue the FHCC beyond the 5-year demonstration and whether the model should be replicated elsewhere. While the delays implementing the IT integration area of the Executive Agreement have been largely outside the control of FHCC officials, they may impact FHCC officials’ ability to operate the FHCC as a fully integrated facility. As we recommended in our February 2011 report, we continue to believe that the Secretaries of VA and DOD should strengthen their ongoing efforts to establish the joint IT system capabilities for the FHCC by developing plans that include scope definition, cost and schedule estimation, and project plan documentation and approval.

The lack of an MTF designation and its related challenges may affect further progress in implementing the FHCC demonstration. The administrative burden and uncertainty resulting from the lack of MTF status may hinder FHCC officials’ ability to efficiently operate the FHCC until DOD clarifies the facility’s status relative to the rest of DOD’s health care system or obtains a legislative change to designate the FHCC as an MTF.

As a reporting tool, the FHCC scorecard has the potential to be useful in tracking performance results over time. However, calculating a monthly summary score for the FHCC scorecard raises concerns about FHCC officials’ ability to convey transparent, meaningful, and accurate performance information to VA and DOD officials and other stakeholders. If the monthly summary score calculations do not account for data collection variation, do not specify a target score(s) that would indicate successful performance, or continue to have errors, then the scorecard’s ability to gauge FHCC performance results is unclear. Until these concerns are addressed, the Secretaries of VA and DOD and Congress will be unable to make fully informed decisions as to whether the FHCC model should continue and whether it should be replicated in other locations.

Recommendations for Executive Action

To ensure that FHCC officials are able to efficiently operate the FHCC and uncertainty regarding the lack of MTF status is resolved, we recommend that the Secretary of Defense seek a legislative change to designate the FHCC as an MTF.

To ensure that the plan to report on FHCC performance results is transparent and provides meaningful information that can assist VA and DOD leadership and Congress in decision making with regard to the future of the FHCC or other VA/DOD integration efforts, we recommend that the Secretaries of Veterans Affairs and Defense direct FHCC leadership to conduct further evaluation of the scorecard reporting tool and its methodology and make revisions that will better ensure the transparency and accuracy of the information reported.

Agency Comments and Our Evaluation

DOD and VA each provided comments on a draft of this report. In its comments, DOD concurred with one of our two recommendations to the Secretary of Defense. (DOD’s comments are reprinted in app. I.) VA, in its comments, generally agreed with our conclusions and concurred with our recommendation to the Secretary of VA. (VA’s comments are reprinted in app. II.) In addition, both VA and DOD provided technical comments which we have incorporated as appropriate.

DOD agreed with our finding that the lack of an MTF designation for the FHCC has posed some challenges and confusion; however, the department did not concur with our recommendation to the Secretary of Defense to seek a legislative change to designate the FHCC as an MTF. Rather than seek a legislative change as we recommended, DOD stated that it will consider seeking legislative authorization for the use of personal services contracts at the FHCC—one of the challenges we discuss in this report. In its response, DOD stated that it anticipates that as the FHCC stabilizes and matures, the confusion among employees and providers at the FHCC due to the lack of an MTF designation will dissipate and that each of the challenges described in the draft report has been addressed through workarounds. We disagree with DOD’s reasoning and maintain that our recommendation should be implemented in order to eliminate the need for the current workarounds and to address any future problems arising from the lack of an MTF designation for the FHCC. Since the FHCC is now providing the services that the NHCGL once did, it should have the same MTF designation the NHCGL had in order to carry out its work as efficiently as possible. Eliminating the need for workarounds could free staff time and contribute to increased efficiency of patient care and operations at the FHCC. In addition, if the FHCC model of collaboration between VA and DOD is replicated elsewhere, the same workarounds will have to be implemented in order to overcome the lack of an MTF designation. If our recommendation were implemented, it would set a precedent for future VA and DOD integrations and help make the integration process smoother.

VA and DOD concurred with our recommendation to the Secretaries of Veterans Affairs and Defense to direct FHCC leadership to conduct further evaluation of the scorecard reporting tool and its methodology, and make revisions that will better ensure the transparency and accuracy of the information reported. In its comments, VA describes the department’s recent actions to implement this recommendation. VA acknowledged that the varying reporting timelines for performance measures resulted in an artificially low monthly summary score in some months when using the original methodology. VA stated, in its comments, that it has changed the calculation process for the scorecard’s monthly score to address this issue. Specifically, FHCC staff will populate the scorecard with a score for each measure every month using either data acquired that month, or the most recent available data for those measures. VA states that this will allow for a more accurate comparison of performance from month to month.

We are sending copies of this report to the Secretary of Defense, Secretary of Veterans Affairs, and appropriate congressional committees. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact Debra A. Draper at (202) 512-7114 or draperd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Comments from the Department of Defense

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Marcia A. Mann, Assistant Director; Jill K. Center; Kaycee M. Glavich; E. Jane Whipple; and Malissa G. Winograd made key contributions to this report. Lisa A. Motley provided legal support and Jennie F. Apter assisted in the message and report development.