Defense Health Care Reform: DOD Needs Further Analysis of the Size, Readiness, and Efficiency of the Medical Force
Published: 2016-09-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD initiated the Study to address perceived weaknesses within the Military Health System and to leverage advances in civilian business practices. The National Defense Authorization Act for Fiscal Year 2015 included a provision for DOD to submit the Study to the congressional defense committees and for GAO to review the Study. DOD submitted its study in February 2016. This report assesses, among other things, the extent to which the Study followed an approach that is consistent with relevant generally accepted research standards and utilized key practices for estimating cost savings. GAO compared the Study with generally accepted research standards that were developed by reviewing research literature and DOD guidance and with key practices derived from cost-estimating guidance.
section_title: What GAO Found
paragraphs: The Department of Defense's (DOD) approach in its Report on Military Health System Modernization (the Study) did not consistently follow relevant generally accepted research standards for research design and execution. While the Study's recommendations position DOD, over time, to take actions to improve the effectiveness and efficiency of the Military Health System, GAO found a number of shortcomings, including the following:
The Study did not fully mitigate limitations identified in its analysis of the required number of active-duty and civilian medical personnel. For example, the Study did not explain how known issues with the military services' workforce models affected the results of its requirements analysis. Without addressing such limitations, DOD will not have a full assessment of its medical workforce needs.
The Study did not sufficiently identify or mitigate limitations concerning its assessment of the requirements necessary to maintain the skills of active-duty medical providers. For example, although there were limitations concerning the accuracy of information on medical providers' workload, the Study did not identify or mitigate these limitations. Having accurate workload information is important to establishing a sound standard for maintaining the clinical skills of medical providers.
The Study established goals for transferring health care from DOD's purchased care network into its own network of hospitals and clinics and for increasing the productivity of active-duty medical providers, but did not develop a strategy explaining how these goals would be achieved. Without such a strategy it remains unclear whether DOD can achieve its goals to transfer health care from the purchased care network into its own network.
DOD's estimated cost savings did not fully utilize key practices for developing such estimates. DOD estimated net annual savings of $366 million from changes to 10 small hospitals and achievement of its goals for recapturing health care and increasing the productivity of active-duty health care providers. However, DOD did not include in its estimate an appropriate level of detail concerning the calculation of estimated savings, all potentially significant costs, or a description of the steps taken by the Study team to assess the reliability of cost data used to develop the estimate. For example, the Study recommended that a number of inpatient facilities be closed, but GAO's analysis found that the Study did not identify estimated costs associated with these changes. As a result, DOD's cost savings estimate did not present a full and accurate picture of possible costs and savings.
section_title: What GAO Recommends
paragraphs: GAO is making six recommendations, including that DOD conduct a new analysis of the required number of active-duty and civilian medical personnel that mitigates known limitations; identify and mitigate limitations regarding the standard for maintaining providers' clinical skills; develop a strategy for achieving its goals for transferring health care to DOD facilities and increasing the productivity of active-duty providers; and, when considering proposed changes to facilities, include in any accompanying cost estimates an appropriate level of detail. DOD concurred with each of GAO's recommendations.

section_title: Letter
paragraphs: Spending on the Military Health System (MHS) has grown substantially  over the past 10 years, increasing from about $24.5 billion in fiscal year  2005 to about $42 billion in fiscal year 2015. To address perceived  weaknesses within the MHS and leverage advances in civilian business  practices, the Department of Defense (DOD) initiated the Report on  Military Health System Modernization (the Study) in 2013. Its two main  goals were to achieve cost savings within the MHS and increase the  readiness of the medical force to support military operations. The Study  recommended a number of interim goals, metrics, and changes to MHS  business practices, but highlighted the need for further analysis of these  issues, making a number of recommendations along those lines. It also  recommended some immediate changes, including changes in  capabilities to 10 of the 24 military hospitals with low numbers of patients,  such as closing inpatient facilities in whole or part for 8 hospitals. The  Study estimated that all of these recommended changes could save $366  million annually. The Under Secretary of Defense for Personnel and  Readiness accepted changes for 8 of the 10 hospitals, and the military  services made further modifications to the recommendations.
Subsequent to DOD initiating the Study, Section 713 of the National  Defense Authorization Act (NDAA) for Fiscal Year 2015 included a  provision for DOD to submit the Study to the congressional defense  committees and also listed nine elements, some with multiple items, to be  incorporated as part of the report. In addition, the NDAA included a  provision for us to review the Study with regard to its methodology and  the data used to support it. DOD provided the Study to the congressional  defense committees in February 2016. We assessed the extent to which  the Study (1) addressed each of the nine elements included in the NDAA;  (2) followed an approach that is consistent with relevant generally  accepted research standards; and (3) utilized key practices for cost  savings estimates.
To address our first objective, we compared Part III of DOD’s submission  to Congress, the data appendix, with the nine elements listed in the  NDAA for Fiscal Year 2015. In evaluating the data appendix, we  considered an element “addressed” if it included all of the items listed in  the NDAA; “partially addressed” if it included some, but not all, of the  items; and “not addressed” if it did not include any of the items within the  element. We discussed the data appendix and our preliminary  observations based on our analyses with officials from the Study team  within the Defense Health Agency.
To address our second objective, we compared the Study with selected  generally accepted research standards relevant to the Study, such as the  need for clearly stated objectives, reasonable assumptions, and efforts to  assess and address data reliability. These standards were developed by  reviewing research literature and DOD guidance and identifying  frequently occurring, generally accepted research standards that are  relevant for defense studies, and were initially used in a prior review of a  DOD study on mobility capabilities. We determined that these standards  are still current and relevant for the purposes of this report. We selected  those standards that were most relevant to the Study, in particular those  standards related to the “design” and “execution” of a defense study. For  further information on the development and application of these generally  accepted research standards, see appendix I. We also interviewed  officials from each of the Study’s working groups within the Defense  Health Agency and representatives of each of the military services  concerning their analytical and decision-making process. To assess  DOD’s efforts to address data reliability, we compared the Study with  selected principles from the standards that, among other things, state that  it is good practice for a study to describe how researchers determined the  data used were valid and reliable for a study’s purposes and to state and  adequately explain the limitations, if any, of the data used to support it. To  conduct this comparison, we (1) reviewed the Study for passages in  which data reliability is discussed; (2) utilized a structured data-collection  instrument to query Study officials regarding their efforts to assess the  reliability of data used in the Study; and (3) systematically searched and  reviewed work papers provided by the Study team to identify descriptions  of how study-team members gathered and assessed the reliability of the  data used in their analyses. We did not independently assess the  reliability of the source data that DOD used in developing the Study.  Rather, as part of our examination of the Study’s adherence to the  relevant generally accepted standards for defense research, we asked  DOD officials what steps they took to assess the reliability of the data  used in developing their respective sections of the final report, and we  reviewed supporting documentation provided by the officials to identify  any known data reliability issues.
To address our third objective, we compared the 2016 Study’s cost  savings estimates for the changes to 10 military hospitals recommended  in the Study and the related achievement of the Study’s goals for  transferring health care to DOD’s own facilities and increasing active-duty  provider productivity with key practices we derived from cost-estimating,  budgeting, and related guidance. The practices call for, among other  things, including all significant costs and assessing the reliability of the  data used to develop the estimate. We discussed the Study’s estimate  with officials from the Study team within the Defense Health Agency.
To address all of our objectives, we met with representatives from  relevant organizations across DOD, including from the Office of the  Assistant Secretary of Defense (Health Affairs), the Defense Health  Agency, and representatives of the medical commands and agencies of  the Army, Navy, and Air Force.
We conducted this performance audit from October 2015 to September  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Overview and Structure of the MHS
paragraphs: The MHS is a complex organization that provides health services to  almost 10 million beneficiaries across a range of care venues, including  its own direct care system in hospitals and clinics, through its purchased  care network of private providers, and on the battlefield in support of  contingency operations. Responsibility for the delivery of care is shared  among the Office of the Assistant Secretary of Defense (Health Affairs),  the military services, and the Defense Health Agency. The Office of the  Assistant Secretary of Defense (Health Affairs) reports to the Under  Secretary of Defense for Personnel and Readiness, who in turn reports to  the Secretary of Defense. The Army and Navy medical commands and  the Air Force medical agency report through their service chiefs to their  respective military department Secretaries and then to the Secretary of  Defense. The Office of the Assistant Secretary of Defense (Health Affairs)  manages the Defense Health Program appropriation, which funds the  service medical departments. However, the military treatment facilities,  including hospitals and clinics, are under the direction and control of the  services, which maintain the responsibility to staff, train, and equip those  commands to meet mission requirements. The MHS collaboratively  develops strategy to meet policy directives and targets. The service  components and the Defense Health Agency are responsible for  execution. Figure 1 depicts the MHS organizational structure.
section_title: Overview of the Military Health System Modernization Study’s Findings and Recommendations
paragraphs: The Study was chartered by the Assistant Secretary of Defense (Health  Affairs) and led by a team including representatives from that office, as  well as from the Defense Health Agency and from the medical commands  and agencies of the Army, Navy, and Air Force. Officials were organized  into four working groups that collectively made up the Study team. Study  officials told us that the majority of the Study’s work was completed in  early 2013, but DOD did not finalize the Study until early 2015, and it did  not submit the Study to the congressional defense committees until  February 2016. The Study included eight recommendations for further  study and analysis in a wide range of areas, such as defining medical  force readiness. The areas addressed in the Study included the following:
Medical Readiness Personnel Requirements: The Study team sought  to determine the number of active-duty uniformed medical personnel  needed to support military operations.
Health Care Demand: The Study team analyzed the demand for  health care in geographical medical markets both within DOD’s direct  care system of hospitals and clinics and from its purchased care  network of private providers. The Study developed goals for the  transfer of health care from the private sector to DOD’s own facilities  both to increase the efficiency of its spending by making better use of  active-duty providers and to provide sufficient workload for them to  maintain their skills.
Provider Currency Requirements: The Study team sought to develop  a metric to assess the currency of medical personnel—that is, an  assessment of the maintenance of clinical skills. Currency, in turn,  supports the readiness of medical personnel, meaning their ability to  provide medical support during operations. However, the Study team  was unable to agree upon a clinical currency measure, and instead  adopted a productivity metric as a proxy for currency. The productivity  metric aims to measure active-duty providers’ workload level  measured in Relative Value Units, a measure of health care  production. The accompanying goal is for active-duty specialist  providers to achieve 40 percent of the median productivity of  specialist private-sector providers as defined by a major medical  survey.
Provider Allocation Model: The Study team developed a model that  matched uniformed active-duty medical providers with demand for  health care within geographic medical markets, with the assumption  that the aforementioned goals for the transfer of health care from the  purchased network of private providers to DOD’s facilities and  increasing the productivity of active-duty providers will be achieved.  This process was intended to demonstrate over- or undersupply of  providers in certain specialties. The analysis found that for some  specialties, there was insufficient demand or other constraining  factors that prevented the matching of all providers to markets. For  example, based on the Study’s assumptions regarding the transfer of  health care to DOD facilities and increasing active-duty provider  productivity, 32 orthopedic surgeons could not be matched to medical  markets, based on 2013 data.
Primary-Care Analysis: The Study examined the ratio of enrolled  beneficiaries to primary-care providers, and found that the MHS was  close to its goal of 1,100 enrollees per primary-care provider.
Overseas Hospital Analysis: Due to a variety of cultural, logistical, and  readiness concerns, the Study recommended no change to overseas  military hospitals. Officials told us that this was the primary issue of  contention within DOD due to concerns surrounding the potential  need for these facilities in support of operations.
The Study also included a facility analysis of 24 small hospitals within the  MHS to determine their future medical capabilities, such as whether they  should continue to have inpatient facilities or birthing centers. The Study  recommended changes for 10 of the 24 hospitals, including closing  inpatient services at 8 of them. The Under Secretary of Defense for  Personnel and Readiness approved proposed changes for 8 of the 10  facilities.
The Study estimated annual net cost savings of $366 million in fiscal year  2012 dollars for the recommended changes to the 10 affected hospitals  and for achieving its goals for transferring health care to DOD facilities  and increasing active-duty provider productivity. This estimated cost  savings included about $132 million of savings associated with reductions  in civilian Full Time Equivalents (FTE), $333 million of savings associated  with decreased purchased care costs, and $98 million in increased  nonlabor costs. Prior to the release of DOD’s report to Congress, the  Navy completed changes to two of the hospitals and retained, though  reduced, inpatient services at the third. The Army also approved selected  changes at three of its facilities.
section_title: DOD’s Study Addressed Four Elements, Partially Addressed Four Elements, and Did Not Address One Element in the 2015 National Defense Authorization Act
paragraphs: In the Report on Military Health System Modernization (the Study), DOD  included information that addressed four of the elements listed in section  713 of the NDAA for Fiscal Year 2015, partially addressed four elements,  and did not address one element. For example, the Study addressed  the element concerning the extent to which DOD consulted with  community hospitals to determine their ability to absorb additional  patients by discussing with local TRICARE experts. Similarly, DOD  addressed the element concerning the recommendations of the Secretary  to restructure or realign military medical treatment facilities by including a  list of recommended changes to 10 such facilities.
However, the Study partially addressed four elements by not addressing  every item of the relevant element or including incomplete information.  For example, the Study partially addressed the element concerning the  impact of recommended changes to military treatment facilities on the  access of eligible beneficiaries to high-quality health care, and  satisfaction with such care. While the Study listed the criteria by which  DOD evaluated the impact of recommended changes on beneficiaries’  access to high-quality health care and satisfaction with such care, it did  not provide further details on the results of its evaluation. Similarly, DOD  partially addressed the element that listed 14 items concerning hospitals  considered under the Study from 2006 to 2012. While the Study provided  information addressing 11 of these items, it did not provide full information  for the remaining 3, such as the omission of budgeted funding levels for  Navy facilities from 2006 through 2009. The Study did not address the  element concerning the impact that proposed changes to military medical  treatment facilities would have on timely access by local civilian  populations to health care services. While the Study included data on  civilian hospital capabilities, it did not explain what impact, if any, the  proposed changes would have on access to such facilities. Appendix II  provides further information on our analysis of the extent to which the  Study addressed the nine elements listed in Section 713 of the NDAA for  Fiscal Year 2015.
Six of the nine elements required a single response, such as an  explanation or analysis. The remaining three elements listed multiple  items, including two elements that required information concerning all of  the hospitals considered under the Study. Although DOD assessed 24  hospitals in its Study, it chose to include information in these two  elements that pertained only to the 8 hospitals for which the Secretary of  Defense approved the recommended changes. Neither the Study nor the  appendix provided a full explanation as to why not every aspect of the  required items was included. The Study appendix noted in one instance,  for the item concerning mental-health availability at training centers, that  there were limited data available.
According to senior Study team officials, DOD compiled the data required  by Section 713 after the Study was finalized in May 2015. Further,  officials stated that, in general, the compilation of these data did not  inform the Study’s analysis, findings, or recommendations, because DOD  had initiated the study on its own initiative prior to the enactment of the  NDAA for Fiscal Year 2015. Although Study team officials stated that  they independently analyzed some of the same data elements as those  listed in the NDAA, they did not examine all such elements. However, for  the data elements they did examine, they may not have examined the  exact attributes specified in the NDAA.
section_title: DOD’s Study Did Not Consistently Follow Relevant Generally Accepted Research Standards
paragraphs: Our analysis of the Study found that it followed some, but not all, of the  relevant generally accepted research standards for research design and  execution. Such standards contribute to a sound and complete study.  According to these standards, a study’s design should include, for  example, establishing the objectives, scope, and methodology, and  identifying study assumptions. Successful execution involves ensuring  that the methodology is carried out as planned and ensuring that data  used in the study are sufficiently valid and reliable for the study’s  purposes. (See app. I for further information on the generally accepted  research standards that we used to conduct our assessment.)  Specifically, we found the following:
The Study’s objectives, scope, and methodology were generally  consistent with relevant accepted research standards.
The Study did not sufficiently mitigate limitations concerning its  analysis of the required number of active-duty medical and civilian  medical personnel.
The Study included a limited assessment of clinical currency—the  requirements necessary to maintain the skills of active-duty medical  providers.
The Study established goals for the transfer of health care from  DOD’s purchased care network into its own network of hospitals and  clinics and for increasing the productivity of active-duty medical  providers without a strategy explaining how these goals would be  achieved.
The Study’s provider allocation model did not reflect differences  among the military services.
The Study did not describe steps taken to ensure the reliability of the  data supporting its analyses and recommendations.
In general, Study officials, including those from the various working  groups, stated that they were limited in their analysis by the time available  to them. Study officials told us that the majority of the Study’s work was  completed in early 2013, but DOD did not finalize the Study until early  2015, and did not submit it to the congressional defense committees until  February 2016.
The Study’s recommendations for future work and analysis position DOD  to continue important efforts to modernize the MHS. However, without  consistently following generally accepted research standards, DOD’s  work on the Study’s recommendations may not be fully sound and  complete. By addressing the areas that we identified in our review, DOD  can strengthen its future efforts.
subsections: section_title: Objectives, Scope, and Methodology Were Generally Consistent with Relevant Accepted Research Standards
paragraphs: The Study’s objectives, scope, and methodology were generally  consistent with relevant accepted research standards for the design of a  study. Generally accepted research standards state that objectives  outline what a study is intended to accomplish; a study’s scope defines  the subject matter that will be assessed and reported on, and is directly  tied to the objectives; and a study’s methodology should be consistent  with and address the objectives.
The Study had five objectives that defined the subject matter to be  addressed, and all of these related to DOD’s high-level goals for the  Study of ensuring the readiness of the medical force and creating a more  efficient MHS. The Study’s objectives generally corresponded to the goals  in the Study’s Charter document. However, two goals stated in the  Study’s Charter were not addressed in these objectives or in the Study’s  execution. Specifically, the Study did not address the goals to finalize  development of an updated joint casualty model, which would provide a  cross-service assessment of the need for medical support in support of  operations, and to develop recommended civilian medical force levels.  (The efforts by the Study team to address the Study’s objectives and  Study’s Charter goals are discussed later in this report.) Although the  Study did not address all of its Charter’s goals, it provided new  information in some areas, such as currency of active-duty providers,  which DOD can use to develop future analysis.
The Study’s scope and methodology were linked with the objectives,  consistent with generally accepted research standards. The Study’s  scope included all medical personnel and facilities that deliver direct  patient care in the Defense Health Program–funded treatment facilities.  The Study’s methodology involved various analyses by the working  groups that were structured around each of the five objectives. The  Study’s work was carried out by corresponding working groups, overseen  by an overarching group. The Study also provides a brief discussion of  “risks and mitigation strategies” comprising a selected list of six changes  in the environment or underlying assumptions that could affect the  Study’s recommendations and an accompanying potential alternative  mitigation strategy to address each change.
section_title: The Study Did Not Fully Mitigate Limitations Concerning Its Analysis of the Required Number of Active-Duty and Civilian Medical Personnel
paragraphs: The Study presented data on the size and composition of the wartime  active-duty medical force, also known as the “operational force.”  However, the data were largely drawn from service models with known  problems and not based on new analysis conducted by the Study team.  To estimate the size of the operational force, the Study presented the  sum total of the services’ individual estimates of their needs. These  estimates are produced by service-specific models and planning tools  that examine a common set of defense planning scenarios. However, the  Study highlighted differences in approach between the services. For  example, the Army chose to use its own estimating tool because the Joint  Medical Planning Tool was not accredited at the time of the Study. In  contrast, the Navy chose to use an early, unaccredited version of the  Joint Medical Planning Tool. Similarly, while the Army used the Joint  Integrated Contingency Model to develop its casualty estimates, which  are an input into the various planning tools, the Navy used four separate  models, including two prototypes. The Air Force does not use casualty  estimates, but rather an estimate of the number of likely evacuees from  theater. The Study noted that a separate independent analysis conducted  by the Joint Staff correlated with the services’ estimates for the  operational force. In addition, Study team officials stated that the services’  processes for developing their “generating” and “institutional” forces,  which include personnel responsible for issues such as graduate medical  education and headquarters staff functions, were not shared with the  Study team. Previous reviews of the necessary size of the medical force  have highlighted the differences between analyses based on service- generated estimates and analyses based on DOD-wide estimates.  Specifically, during the 1990s, two analyses based on DOD-wide  estimates found the services had excess medical personnel relative to  their needs, while one analysis based on service-generated estimates  found the services needed nearly all of their medical personnel to meet  their needs.
Generally accepted research standards state that limitations and  constraints, which are generally found in studies, should be identified and  reasonably mitigated. While the Study clearly outlined the limitations of  its service-oriented approach, the Study did not fully mitigate them, such  as by explaining how these limitations affected the results of the analysis.  Officials with the working group that addressed readiness requirements  stated that their methodology was limited by the time available for  analysis. Specifically, the officials stated that they knew they would be  unable, in the time allowed, to conduct a full, independent analysis of the  number of active-duty medical personnel necessary to support  contingency operations, and that they therefore relied upon estimates  from the military services. Officials stated that it would be feasible to  conduct an independent analysis of the required number of active-duty  medical personnel if there were greater information sharing from the  services and more time to complete such an analysis. While we recognize  the time constraints faced by the Study team, addressing these issues  would provide more meaningful results as DOD considers further  changes to the MHS. Absent such efforts to address these limitations,  DOD will not have a full assessment of its medical workforce needs.
In addition, the Study’s Charter included a goal to determine civilian  personnel levels, but the Study did not include such an assessment in its  scope. Civilians represent a significant proportion of the MHS workforce.  For example, in 2015, civilians represented 61 percent of the Army’s  medical positions. Civilian personnel levels also have a direct effect on  active-duty personnel levels, and DOD policy requires the services and  defense agencies to determine the appropriate mix of manpower (military  and civilian) and private-sector support. DOD’s Medical Readiness  Review, an outgrowth of the 2006 Quadrennial Defense Review, found  that efficiencies could be made by converting military billets to civilian  positions, although a prohibition on such conversions was later put in  place. Without addressing the role of civilians in the MHS, the Study’s  analysis does not present a complete picture of DOD’s needs.
section_title: The Study Included a Limited Assessment of Clinical Currency
paragraphs: The Study included a limited assessment of clinical currency that did not  sufficiently identify or mitigate several limitations and constraints. As  noted earlier, the Study team was unable to agree upon a measure of the  clinical currency of active-duty medical personnel—that is, an assessment  of the maintenance of skills. Currency, in turn, supports the readiness of  medical personnel, meaning their ability to provide medical support during  operations. The Study stated that the Study team attempted to identify an  industry standard for clinical currency, but none existed. In addition, a  Study official stated that the working group had a limited time frame to  complete its work. The Study team instead adopted a productivity metric  as a proxy for currency, along with an accompanying goal for active-duty  specialist providers to achieve 40 percent of the median productivity of  private-sector specialist providers as defined by a major medical survey.
However, some Study team officials expressed doubts concerning the  accuracy of provider productivity data used to develop the proxy measure  for clinical currency. Specifically, officials stated that they believe that the  MHS as a whole does not accurately record Relative Value Units,  meaning that the MHS does not accurately record providers’ workload.  Further, a senior Study team official stated the military services and  individual military treatment facilities differ in how they record this  information, and, due to these differences, the Study team adjusted the  Relative Value Unit figures for a number of hospitals, sometimes with a  margin of error of up to 50 percent. Generally accepted research  standards state that limitations and constraints should be identified and  reasonably mitigated. However, the limitations in the accuracy of  information concerning providers’ workload were not identified or  mitigated in the Study. Having accurate workload information is important  to establishing a sound standard for maintaining the clinical skills of  medical providers.
Further, in comparing the productivity of active-duty medical providers  with civilian medical providers, the Study reduced the amount of time  active-duty providers are available to practice medicine because of the  “readiness cost” of time that they must devote to military-specific duties  and requirements, such as physical training, military training, and federal  holidays. The Study team’s approach relied on subject-matter experts to  estimate the time active-duty providers had available to practice  medicine, and a Study official said there were no reliable data concerning  the average number of clinical hours worked by active-duty providers.
Other approaches, such as time studies of active-duty providers, are  more rigorous, but the Study official stated that the working group’s  approach was limited by time constraints.
The Study contained a recommendation to further study the costs of  medical force readiness, and DOD is currently working to establish a  metric for medical force readiness. However, without the benefit of more  rigorous analysis concerning active-duty providers’ workload and the time  they must devote to military-specific duties and responsibilities, DOD will  not have a sound and accurate standard for maintaining the clinical skills  of medical providers.
section_title: The Study Established Goals for Transferring Health Care into Its Own Network of Hospitals and Clinics and Increasing the Productivity of Providers without a Strategy Explaining How These Goals Would Be Achieved
paragraphs: The Study, in developing its findings and recommendations, assumed  that DOD would be successful in its goals for transferring health care from  patients being treated in DOD’s purchased private-sector network into its  own military treatment facilities, as well as in achieving related goals for  improving provider productivity. The Study recommended the  establishment of associated goals, and DOD adopted such goals in  March 2014. However, this approach was not consistent with the  generally accepted research standard that assumptions should be  reasonable. Specifically, the Study’s assumptions regarding its goals for  transfer of health care to DOD’s facilities and increasing active-duty  provider productivity are unsupported because DOD has not developed  an accompanying strategy describing how these goals can be achieved.
Our prior work has identified six selected leading practices of effective  federal strategic planning, which we derived in part from the Government  Performance and Results Act (GPRA), as updated by the GPRA  Modernization Act of 2010, associated guidance, and our prior work.  Specifically, these leading practices are to: (1) define the mission and  goals, (2) define strategies that address management challenges and  identify resources needed to achieve goals, (3) ensure leadership  involvement and accountability, (4) involve stakeholders, (5) coordinate  with other federal agencies, and (6) develop and use performance  measures. Study team officials told us that, notwithstanding the absence  of a strategy, transfer of health care to DOD facilities is a priority and that  such goals were modest and achievable. However, the Study set out  goals for transferring health care into its own facilities for large groups of  beneficiaries, such as TRICARE for Life beneficiaries, for whom, unlike  active-duty military and their families, the MHS has few tools to  encourage their use of military treatment facilities.
In addition, transfer of health care into DOD facilities has been an MHS  goal for several years, but limited progress has been made. For example,  the Congressional Budget Office reported in 2014 that most of the  growth in outpatient utilization of health care services within the MHS has  occurred within the purchased care system, while military hospitals and  clinics have maintained a relatively constant workload since 2005.  Similarly, it is uncertain that DOD can achieve its related productivity  goals without an accurate means for measuring providers’ workload, as  discussed earlier. Without a strategy that incorporates leading practices, it  remains unclear whether or how DOD can achieve its goals to transfer  health care from the purchased care network into its own network of  facilities.
section_title: The Study’s Provider Allocation Model Did Not Reflect Differences among the Military Services
paragraphs: The Study included a provider allocation model that identified the medical  specialties needed with anticipated health care demand within specific  geographic medical markets. The model’s results highlighted specialties  for which the MHS has too many medical specialists relative to  anticipated market demand for various health care services and other  specialties for which the MHS has too few physicians to accommodate  market demand. Generally accepted research standards state that  models should, among other things, represent a reasonably complete  range of conditions and be appropriate for the intended purpose.  However, an important aspect of the provider model did not reflect the  standards in that it unreasonably assumed that medical providers from  the different services were interchangeable. Specifically, the Study’s  model for allocation of specialist physicians based on anticipated health  care demand in geographic markets did not account for the military- service ownership of the relevant hospitals and the physicians’ military  service. Instead, it matched the number of physicians across all services  within a given specialty, such as orthopedic surgery, to anticipated health  care demand in geographic markets, regardless of the service ownership  of the medical treatment facility in that market. While such arrangements  do exist in “multiservice markets,” such as the National Capital Region  and San Antonio, they are less common elsewhere. The model could  potentially provide results that do not reflect MHS business practices in  the real world. For example, it could suggest that there are sufficient  physicians to provide a given type of specialty care to a particular  geographic medical market, when in reality there are not sufficient  physicians from that medical market’s primary military service to assign.  Currently, active-duty providers generally practice within their own  services’ hospitals in a team context unique to their military service. The  Study emphasized the view that it is important for physicians to practice  medicine within this team context of other specialists and nurses from the  same service, which can be seen within the services’ different  configurations of medical personnel in a deployed setting.
CNA independently assessed one part of DOD’s model. However, its  assessment included only the mechanics of the model and not an  examination of the supporting data used in the model. The Study stated  that the Study team relied on military-service representatives to verify  these data. In addition, CNA noted that alternative approaches could yield  different results depending on assumptions concerning transfer of health  care to DOD facilities and increased active-duty provider productivity. The  Study asserted that its modeling results were reasonable given the time  constraints of the Study. The Study recommended further work to refine  and formalize the model and use it to inform decisions concerning the  delivery of health care services. However, unless DOD addresses  differences between the services and the effects of the model’s practical  application, the model’s usefulness will be limited and could lead to  impractical decisions.
section_title: The Study Did Not Describe Steps Taken to Ensure the Reliability of the Data Supporting Its Analyses and Recommendations
paragraphs: The Study and the data appendix did not describe steps that the Study  team took to ensure that the data used to support the Study and data  appendix were sufficiently reliable for the Study’s purposes. Generally  accepted research standards state that the data used in a study should  be valid for the intended research purpose; in other words it represents  what it purports to represent. In addition, documenting a study’s  analytical and decision-making process helps to ensure the accuracy,  completeness, and credibility of study results. However, we found that the  study presented limited information on its efforts to assess data reliability.  For example, the Study’s data appendix did not document the source of  its data in several instances. Specifically, the data source is provided for 9  of 15 data metrics, but not for the other 6 metrics.
Because the Study presented limited information on the Study team’s  efforts to assess the reliability of the data used, we provided each working  group’s lead official with a standard set of questions concerning their  efforts to ensure data reliability. Specifically, we asked the working group  leads which data sources they used in their respective analyses. Their  responses indicated that they used five databases and one data set that  were not mentioned in the main body of the Study or the data appendix.  We questioned the working group leads concerning steps taken to ensure  data reliability, such as whether they consulted with the source or owner  of the data to determine what methods were used to ensure reliability or  whether they took other steps to ensure reliability, such as conducting  their own assessment or reliance on existing assessments. Working  group officials stated that they generally relied on the reliability processes  of the owners of the data to ensure the data provided were reliable. For  example, two of the working groups stated that they used databases that  are commonly used within the MHS, and that these databases are subject  to oversight reviews and periodic validity checks. An official from one  working group stated that the group also conducted reviews of relevant  data to check for completeness of records related to inpatient care and  found missing data. The working group developed adjustment factors and  applied them by market and provider specialty to impute missing data  where possible. An official from another working group stated that the  group used data provided by contractors, and that DOD’s subject-matter  experts routinely discuss report content with the contractors as a way to  verify the validity and reliability of the data provided as part of the  performance assessment of the contract. In addition, we interviewed  service officials regarding the accuracy and reliability of the data used in  the Study, and officials from the Navy disputed the accuracy of the  Study’s reported physician productivity data. Army officials stated that  they had some differences in data interpretation based on their own  analysis, which led them to some different recommendations.
When asked if there were any limitations on the use of specific data  sources, working group officials reported to us some limitations that were  not specifically described in the main body of the Study or data appendix.  For example, one working group official mentioned that military treatment  facility bed counts and ambulatory surgery counts were not consistently  reported, though officials stated that these data did not appear to have a  significant effect on the group’s findings. Another limitation concerned the  consistency of data from the Medical Expense and Performance  Reporting System affecting the assessment of a primary-care provider’s  availability. An official noted that as a result of this limitation, the final  analysis of primary-care capacity was curtailed and specific  recommendations were limited.
A systematic review of over 300 supporting work papers provided to us by  Study officials found multiple examples of known data or analysis  limitations that, if the Study was adhering to generally accepted research  standards, would have been conveyed in the Study. For example, one of  the work papers noted data reliability issues with certain overseas-related  data which were not noted in the report. One work paper noted some  limitations in a data source used in the study that were not discussed in  the report, and another work paper mentioned having to manually adjust  some data metrics, which was not noted in the Study. Conveying data or  analysis limitations is important so that decision makers relying on the  Study can fully understand the quality and soundness of the research.  Without sufficiently demonstrating the accuracy, reliability, and limitations  of the data upon which DOD makes policy changes, decision makers do  not have assurance that the Study’s findings and recommendations are  based on sound information.
section_title: DOD’s Estimate of Cost Savings from Changes to Small Hospitals Did Not Fully Reflect Key Practices
paragraphs: DOD’s estimated cost savings for the recommended changes to 10 small  hospitals and the related achievement of goals related to transfer of  health care to DOD facilities and increased active-duty provider  productivity included a net estimate of savings and costs, but did not fully  reflect key practices for cost and cost savings estimates. For example,  DOD included estimated savings from three categories—reductions in  civilian personnel, net decreases in purchased care, and net increases in  nonlabor costs—and listed general sources of potential savings.  However, the Study did not fully reflect key practices we derived from  cost-estimating, budgeting, and related guidance. These key practices  emphasize the importance of an appropriate level of detailed  documentation, the inclusion of all significant costs, and an assessment  of the reliability of the data used to develop the estimate, among others.
First, the Study did not provide an appropriate level of detail concerning  the calculation of the estimated cost savings or the assumptions upon  which they were based. For example, the Study’s cost savings estimate  did not include details concerning the number of military personnel who  would be moved from the 10 affected facilities to achieve the savings  associated from transfer of health care to DOD facilities and increased  active-duty provider productivity. In addition, of the total estimated  savings, $333 million was associated with changes that were based on an  assumption concerning DOD’s ability to achieve its goals related to  transfer of health care to DOD facilities and increased active-duty  provider productivity without stating a strategy to reach those targets.  Absent a clear and likely achievable strategy, the predicted cost savings  associated with transfer of health care to DOD facilities and increasing  active-duty provider productivity are questionable.
Second, the Study’s predicted net savings estimate did not include all  potentially significant costs. Specifically, its cost savings estimate did not  address the effect of the following costs:
Costs associated with a reduction-in-force of civilian FTEs: The Study  estimated savings of $132 million associated with the elimination of  1,324 civilian FTEs across the MHS due to the aforementioned  assumption of greater productivity of active-duty providers. However,  we have previously reported that DOD’s efforts to reduce civilian  FTEs do not always result in planned decreased costs of a  proportionate magnitude to the decrease in FTEs. For example, in  2016, we found that across DOD, from fiscal years 2012 through  2016, civilian FTEs declined by 3.3 percent, but civilian personnel  costs declined by only 0.9 percent, adjusted for inflation.
Onetime costs to realign active-duty providers to other hospitals: The  Study recommended changes that would realign active-duty providers  to other facilities. However, movement of these providers, the  associated active-duty medical professionals with whom they work in  a team setting, and their families will incur costs such as permanent  change of station expenditures. Study officials told us that they did not  include these costs because permanent change of station costs vary  widely and final decisions concerning where personnel would be  relocated rest with the military services. We previously reported on the  costs of permanent change of station moves and found that these  moves, on average, cost about $6,700 per servicemember in 2014.
Construction costs, if applicable, at facilities losing or gaining active- duty providers: The Study recommended that a number of inpatient  facilities be closed or converted to ambulatory surgery centers, but  our analysis found that the Study did not identify estimated costs  associated with these changes. Construction costs also may be  applicable at facilities gaining active-duty providers, but Study officials  stated that execution of such changes would be the responsibility of  the services. However, without specifically addressing these costs or  justifying their exclusion, the Study’s cost savings estimate leaves  open the question of whether potential savings could be reduced by  such costs.
Third, DOD’s savings estimate did not describe the steps taken by the  Study team to assess the reliability of cost data used to develop the cost  savings estimate. As discussed earlier in this report, the Study did not cite  the data sources from which the cost estimates were drawn, and did not  discuss what limitations, if any, such data might have had for the  purposes to which they were applied.
As a result of these weaknesses, DOD may not have accurately  estimated the net savings that might result from its proposed hospital  changes. As DOD considers additional changes to its network of military  treatment facilities, decision makers do not have a full and accurate  assessment of all relevant costs and savings. Without a full and accurate  assessment, DOD is not in a position to determine whether its efforts to  respond to the Study’s recommendations will result in savings or,  alternatively, in additional costs.
section_title: Conclusions
paragraphs: In initiating the Study, DOD began the process of examining difficult and  complex problems affecting the MHS. The Study’s recommendations for  further analysis and work in a number of areas position DOD, over time,  to explore these areas and take actions to improve the effectiveness and  efficiency of the MHS. However, our analysis of the Study highlighted  shortcomings that merit review and mitigation before DOD takes definitive  action on these recommendations. Specifically, without addressing the  issues discussed in our report, DOD’s work to address the Study’s  recommendations will continue to reflect the various unsupported  assumptions, unmitigated limitations and constraints, and questions  concerning the reliability of data used to develop the Study’s  recommendations that we have highlighted. Similarly, as DOD considers  further changes to its network of facilities and the medical services they  offer, the analysis supporting these decisions will be incomplete without a  full and accurate picture of the true cost of such changes.
section_title: Recommendations for Executive Action
paragraphs: To fully assess the size and composition of the medical force, we  recommend that the Secretary of Defense direct the Assistant Secretary  of Defense (Health Affairs) to conduct a new analysis of the required  number of active-duty and civilian medical personnel that mitigates known  limitations.
To strengthen ongoing efforts to analyze the costs of medical force  readiness and establish clinical currency standards, we recommend that  the Secretary of Defense direct the Assistant Secretary of Defense  (Health Affairs) to take steps to identify and mitigate limitations regarding  the standard for maintaining providers’ clinical skills, including improving  the accuracy of information concerning providers’ workload and  conducting an analytically rigorous calculation of active-duty providers’  time devoted to military-specific responsibilities.
To help achieve DOD’s goals for transferring health care into its own  facilities and increasing the productivity of active-duty medical providers,  we recommend that the Secretary of Defense direct the Assistant  Secretary of Defense (Health Affairs) to develop a strategy for achieving  these goals that reflects the leading practices of effective federal strategic  planning.
To strengthen ongoing efforts within DOD to address the Study’s  recommendations to use the provider model outputs to inform execution  of health care delivery and to refine the model for future use, we  recommend that the Secretary of Defense direct the Assistant Secretary  of Defense (Health Affairs) to modify DOD’s model to reflect the military  service of the physicians and military treatment facilities included in the  model.
To strengthen any future assessments of additional changes to DOD’s  network of military treatment facilities, we recommend that the Secretary  of Defense direct the Assistant Secretary of Defense (Health Affairs) to  take the following two actions: describe steps taken to assess the reliability of data supporting the  assessment, including, at a minimum, the sources of data, data  limitations, and efforts to test data reliability; and  include in any accompanying cost estimates an appropriate level of  detail, all significant costs, and an assessment of the reliability of the  data supporting the cost estimate.
section_title: Agency Comments
paragraphs: In written comments on a draft of this report, DOD concurred with our six  recommendations to improve DOD’s analysis of the size, readiness, and  efficiency of the medical force. DOD’s comments are reprinted in  appendix III.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Deputy Secretary of Defense,  the Under Secretary of Defense for Personnel and Readiness, the  Assistant Secretary of Defense (Health Affairs), the Defense Health  Agency Director, the Surgeon General of the Air Force, the Surgeon  General of the Army, and the Surgeon General of the Navy. In addition,  the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.
section_title: Appendix I: Selection of Generally Accepted Research Standards Relevant for the Military Health System Modernization Study
paragraphs: To guide our review of the Department of Defense’s (DOD) Report on  Military Health System Modernization (the Study), we identified relevant  generally accepted research standards for the design, execution, and  presentation of findings that define a sound and complete study. We  developed these standards by reviewing research literature and DOD  guidance and identifying frequently occurring, generally accepted  research standards that are relevant for defense studies. We initially used  these standards in a prior GAO review of a DOD study on mobility  capabilities. That report includes a list of 36 questions to ask in  assessing the application of generally accepted research standards. To  develop these standards, we reviewed research literature and DOD  guidance and identified frequently occurring, generally accepted research  standards that are relevant for defense studies that define a high-quality  or methodologically sound and complete study. The following were our  sources for these standards:  GAO, Government Auditing Standards: 2011 Revision,  GAO-12-331G (Washington, D.C.: December 2011);  GAO, Designing Evaluations, GAO-12-208G  (Washington, D.C.:  January 2012);   GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.:  February 2004);   RAND Corporation, RAND Standards for High-Quality Research  and Analysis, CP-413 (5/15) (Santa Monica, CA.: May 2015);   Air Force Office of Aerospace Studies, Analysts Handbook: On  Understanding the Nature of Analysis (January 2000);   Air Force, Office of Aerospace Studies, Air Force Analysis  Handbook, A Guide for Performing Analysis Studies: For Analysis  of Alternatives or Functional Solution Analysis (July 2004);   Department of Defense, DOD Modeling and Simulation (M&S)  Verification, Validation, Accreditation (VV&A), Instruction 5000.61  (Washington, D.C.: Dec. 9, 2009);   Department of Defense, Support for Strategic Analysis, Directive  8260.05 (Washington, D.C.: July 7, 2011);   Department of Defense, Implementation of Data Collection,  Development, and Management for Strategic Analyses, Instruction  8260.2 (Washington, D.C.: Jan. 21, 2003).
We determined that these standards are still current and relevant for the  purposes of this report.
We confined our assessment to the standards related to design (11 of 14  standards) and execution (10 of 15 standards) because the National  Defense Authorization Act for Fiscal Year 2015 mandated we assess the  methodology and data used in the Study. Consequently, we did not  assess DOD’s report against the generally accepted research standards  for presentation of results (7 standards). Of the applicable standards  related to design and execution, we combined and slightly revised the  standards to match the particular analyses used in the study. Some of the  design and execution standards were not relevant to our assessment. For  example, the standards concerning analysis of scenarios and threats are  specific to certain types of defense studies and did not relate to the study  that was the subject of our review. We compared DOD’s approach to the  Study with these selected generally accepted research standards.  Accordingly, we technically reviewed the content of DOD’s report against  the following 18 standards that were relevant to our review:  a.  Objectives, Scope and Methodology: Objectives outline what a  study is intended to accomplish. A study’s scope defines the  subject matter that will be assessed and reported on, and the  scope is directly tied to the audit objectives. A study’s  methodology should be consistent with and address a study’s  objectives.
Is the study’s design clear?
Is the study’s objective clearly stated?
Is the study’s scope clearly defined?
Do the study scope, methodology, and objectives fully  address the study charter and associated guidance?
Was the study plan followed?
Were deviations from the study plan explained and  documented?  b.  Assumptions: Assumptions should be reasonable, consistent,  appropriately varied, and explicitly identified in order to understand  the context of the study’s results. Assumptions define the  parameters for what will be included and how the study is  executed.
Are the assumptions explicitly identified?
Are the assumptions reasonable and consistent?
Are the assumptions varied to allow for sensitivity analyses?  c.  Limitations and Constraints: Limitations and constraints, which are  generally found in studies, should be identified and reasonably  mitigated. Research standards do not call for the elimination of all  limitations, but do call for them to be identified, assessed,  mitigated, and explained.
Are the limitations explicitly identified?
Are the major constraints identified and discussed?  a.  Models: Models should be reasonable ones to consider, represent  a reasonably complete range of conditions, as well as be  appropriate for the intended purpose.
Are the scenarios that were modeled reasonable ones to  consider?
Do the scenarios represent a reasonably complete range of  conditions?
Were the models used to support the analyses appropriate for  their intended purpose? Have they been described and  documented adequately?
Were any modeling limitations identified, explained, and  justified?  b.  Data reliability: The data used in a study should be valid for the  intended research purpose; in other words, they represent what  they purport to represent. Documenting a study’s analytical and  decision-making process helps to ensure the accuracy,  completeness, and credibility of study results.
Were the data used valid for the study’s purposes?
Were the data used sufficiently reliable for the study’s  purposes?
Were any data limitations identified and was the effect of the  limitations adequately explained?
section_title: Appendix II: Comparison of DOD’s Military Health System Modernization Study with Requirements Listed in 2015 National Defense Authorization Act
paragraphs: Table 1 summarizes our assessment of the extent to which the  Department of Defense (DOD) in its Report on Military Health System  Modernization (the Study) addressed each of the required nine elements  listed in Section 713 of the National Defense Authorization Act (NDAA) for  Fiscal Year 2015. Some of the elements required that multiple items be  addressed. We considered an element “addressed” if it included all of the  items listed in the NDAA; “partially addressed” if it included some, but not  all, of the items; and “not addressed” if it did not include any of the items.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Lori Atkinson, Assistant Director;  Rebekah Boone; Patricia Donahue; Mae Jones; Linda Keefer; Carol  Petersen; Adam Smith; Sabrina Streagle; and Adriel Taslim made key  contributions to this report.
section_title: Related GAO Products
paragraphs: Defense Health Care Reform: Actions Needed to Help Ensure Defense  Health Agency Maintains Implementation Progress. GAO-15-759.  Washington, D.C.: September 10, 2015.
Defense Health Care Reform: Actions Needed to Help Realize Potential  Cost Savings from Medical Education and Training. GAO-14-630.  Washington, D.C.: July 31, 2014.
Military Health System: Sustained Senior Leadership Needed to Fully  Develop Plans for Achieving Cost Savings. GAO-14-396T. Washington,  D.C.: February 26, 2014.
Defense Health Care Reform: Additional Implementation Details Would  Increase Transparency of DOD’s Plans and Enhance Accountability.  GAO-14-49. Washington, D.C.: November 6, 2013.
Defense Health Care: Additional Analysis of Costs and Benefits of  Potential Governance Structures Is Needed. GAO-12-911. Washington,  D.C.: September 26, 2012.
Defense Health Care: Applying Key Management Practices Should Help  Achieve Efficiencies within the Military Health System. GAO-12-224.  Washington, D.C.: April 12, 2012.
Military Personnel: Enhanced Collaboration and Process Improvements  Needed for Determining Military Treatment Facility Medical Personnel  Requirements. GAO-10-696. Washington, D.C.: July 29, 2010.