DEFENSE HEALTH 

CARE REFORM

DOD Needs Further 

Analysis of the Size, 

Readiness, and 

Efficiency of the 

Medical Force

Report to Congressional Committees

September 2016

GAO-16-820

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-16-820, a report to 

congressional committees

September 2016

DEFENSE HEALTH CARE REFORM

DOD Needs Further Analysis of the Size, 

Readiness, and Efficiency of the Medical Force

What GAO Found

The Department of Defense’s (DOD) approach in its Report on Military Health 

System Modernization (the Study) did not consistently follow relevant generally 

accepted research standards for research design and execution. While the 

Study’s recommendations position DOD, over time, to take actions to improve 

the effectiveness and efficiency of the Military Health System, GAO found a 

number of shortcomings, including the following: 

• The Study did not fully mitigate limitations identified in its analysis of the 

required number of active-duty and civilian medical personnel. For example, 

the Study did not explain how known issues with the military services’

workforce models affected the results of its requirements analysis. Without 

addressing such limitations, DOD will not have a full assessment of its 

medical workforce needs.

• The Study did not sufficiently identify or mitigate limitations concerning its 

assessment of the requirements necessary to maintain the skills of activeduty medical providers. For example, although there were limitations 

concerning the accuracy of information on medical providers’ workload, the 

Study did not identify or mitigate these limitations. Having accurate workload 

information is important to establishing a sound standard for maintaining the 

clinical skills of medical providers.

• The Study established goals for transferring health care from DOD’s 

purchased care network into its own network of hospitals and clinics and for 

increasing the productivity of active-duty medical providers, but did not 

develop a strategy explaining how these goals would be achieved. Without 

such a strategy it remains unclear whether DOD can achieve its goals to 

transfer health care from the purchased care network into its own network.

DOD’s estimated cost savings did not fully utilize key practices for developing 

such estimates. DOD estimated net annual savings of $366 million from changes 

to 10 small hospitals and achievement of its goals for recapturing health care 

and increasing the productivity of active-duty health care providers. However, 

DOD did not include in its estimate an appropriate level of detail concerning the 

calculation of estimated savings, all potentially significant costs, or a description 

of the steps taken by the Study team to assess the reliability of cost data used to 

develop the estimate. For example, the Study recommended that a number of 

inpatient facilities be closed, but GAO’s analysis found that the Study did not 

identify estimated costs associated with these changes. As a result, DOD’s cost 

savings estimate did not present a full and accurate picture of possible costs and 

savings.

View GAO-16-820. For more information, 

contact Brenda S. Farrell at (202) 512-3604 or 

FarrellB@gao.gov.

Why GAO Did This Study

DOD initiated the Study to address 

perceived weaknesses within the 

Military Health System and to leverage 

advances in civilian business practices. 

The National Defense Authorization 

Act for Fiscal Year 2015 included a 

provision for DOD to submit the Study 

to the congressional defense 

committees and for GAO to review the

Study. DOD submitted its study in 

February 2016. This report assesses, 

among other things, the extent to 

which the Study followed an approach 

that is consistent with relevant 

generally accepted research standards 

and utilized key practices for 

estimating cost savings. GAO 

compared the Study with generally 

accepted research standards that were 

developed by reviewing research 

literature and DOD guidance and with 

key practices derived from costestimating guidance. 

What GAO Recommends

GAO is making six recommendations, 

including that DOD conduct a new

analysis of the required number of 

active-duty and civilian medical 

personnel that mitigates known 

limitations; identify and mitigate 

limitations regarding the standard for 

maintaining providers’ clinical skills; 

develop a strategy for achieving its 

goals for transferring health care to 

DOD facilities and increasing the

productivity of active-duty providers;

and, when considering proposed 

changes to facilities, include in any 

accompanying cost estimates an 

appropriate level of detail. DOD 

concurred with each of GAO’s 

recommendations. 
Page i GAO-16-820 Military Health Modernization

Letter 1

Background 5

DOD’s Study Addressed Four Elements, Partially Addressed Four 

Elements, and Did Not Address One Element in the 2015 

National Defense Authorization Act 9

DOD’s Study Did Not Consistently Follow Relevant Generally 

Accepted Research Standards 10

DOD’s Estimate of Cost Savings from Changes to Small Hospitals 

Did Not Fully Reflect Key Practices 21

Conclusions 23

Recommendations for Executive Action 23

Agency Comments 24

Appendix I Selection of Generally Accepted Research Standards Relevant 

for the Military Health System Modernization Study 27

Appendix II Comparison of DOD’s Military Health System Modernization Study 

with Requirements Listed in 2015 National Defense 

Authorization Act 31

Appendix III Comments from the Department of Defense 34

Appendix IV GAO Contact and Staff Acknowledgments 37

Related GAO Products 38

Table

Table 1: Comparison of DOD’s Military Health System 

Modernization Study with Elements Listed in Section 713 

of the NDAA for Fiscal Year 2015 31

Contents
Page ii GAO-16-820 Military Health Modernization

Figure

Figure 1: Military Health System (MHS) Organizational Structure 6

Abbreviations

DOD Department of Defense

MHS Military Health System

NDAA National Defense Authorization Act

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-16-820 Military Health Modernization

441 G St. N.W.

Washington, DC 20548

September 21, 2016

Congressional Committees

Spending on the Military Health System (MHS) has grown substantially 

over the past 10 years, increasing from about $24.5 billion in fiscal year 

2005 to about $42 billion in fiscal year 2015.1 To address perceived 

weaknesses within the MHS and leverage advances in civilian business 

practices, the Department of Defense (DOD) initiated the Report on 

Military Health System Modernization (the Study) in 2013.2 Its two main 

goals were to achieve cost savings within the MHS and increase the 

readiness of the medical force to support military operations. The Study 

recommended a number of interim goals, metrics, and changes to MHS 

business practices, but highlighted the need for further analysis of these 

issues, making a number of recommendations along those lines. It also 

recommended some immediate changes, including changes in 

capabilities to 10 of the 24 military hospitals with low numbers of patients, 

such as closing inpatient facilities in whole or part for 8 hospitals. The 

Study estimated that all of these recommended changes could save $366 

million annually. The Under Secretary of Defense for Personnel and 

Readiness accepted changes for 8 of the 10 hospitals, and the military 

services made further modifications to the recommendations.3

Subsequent to DOD initiating the Study, Section 713 of the National 

Defense Authorization Act (NDAA) for Fiscal Year 2015 included a 

provision for DOD to submit the Study to the congressional defense 

committees and also listed nine elements, some with multiple items, to be 

incorporated as part of the report.4 In addition, the NDAA included a 

provision for us to review the Study with regard to its methodology and 

 1

Figures in nominal dollars. We determined these figures through analysis of Defense 

Health Program budget documentation. 

2

Department of Defense, Report on Military Health System Modernization: Response to 

Section 713 of the Carl Levin and Howard P. “Buck” McKeon National Defense 

Authorization Act for Fiscal Year 2015 (P.L. 113-291).

3

DOD did not revise its cost savings estimate based on these changes to the Study’s 

recommendations.

4

Pub. L. No. 113-291, Section 713.

Letter
Page 2 GAO-16-820 Military Health Modernization

the data used to support it. DOD provided the Study to the congressional 

defense committees in February 2016.5 We assessed the extent to which 

the Study (1) addressed each of the nine elements included in the NDAA; 

(2) followed an approach that is consistent with relevant generally 

accepted research standards; and (3) utilized key practices for cost 

savings estimates.

To address our first objective, we compared Part III of DOD’s submission 

to Congress, the data appendix, with the nine elements listed in the 

NDAA for Fiscal Year 2015. In evaluating the data appendix, we 

considered an element “addressed” if it included all of the items listed in 

the NDAA; “partially addressed” if it included some, but not all, of the 

items; and “not addressed” if it did not include any of the items within the 

element. We discussed the data appendix and our preliminary 

observations based on our analyses with officials from the Study team 

within the Defense Health Agency.

To address our second objective, we compared the Study with selected 

generally accepted research standards relevant to the Study, such as the 

need for clearly stated objectives, reasonable assumptions, and efforts to 

assess and address data reliability. These standards were developed by 

reviewing research literature and DOD guidance and identifying 

frequently occurring, generally accepted research standards that are 

relevant for defense studies, and were initially used in a prior review of a 

 5

DOD’s Report on Military Health System Modernization (the Study) to Congress includes 

three parts. Part I is the language of section 713 of the NDAA for Fiscal Year 2015; Part II 

is DOD’s Military Health System Modernization Study Team Report; Part III is DOD’s 

detailed responses, including data, requested in Section 713. For purposes of this report, 

we refer to all three parts collectively as the Study.
Page 3 GAO-16-820 Military Health Modernization

DOD study on mobility capabilities.6 We determined that these standards 

are still current and relevant for the purposes of this report. We selected 

those standards that were most relevant to the Study, in particular those 

standards related to the “design” and “execution” of a defense study. For 

further information on the development and application of these generally 

accepted research standards, see appendix I. We also interviewed 

officials from each of the Study’s working groups within the Defense 

Health Agency and representatives of each of the military services 

concerning their analytical and decision-making process. To assess 

DOD’s efforts to address data reliability, we compared the Study with 

selected principles from the standards that, among other things, state that 

it is good practice for a study to describe how researchers determined the 

data used were valid and reliable for a study’s purposes and to state and 

adequately explain the limitations, if any, of the data used to support it. To 

conduct this comparison, we (1) reviewed the Study for passages in 

which data reliability is discussed; (2) utilized a structured data-collection 

instrument to query Study officials regarding their efforts to assess the 

reliability of data used in the Study; and (3) systematically searched and 

reviewed work papers provided by the Study team to identify descriptions 

of how study-team members gathered and assessed the reliability of the 

 6

See GAO, Defense Transportation: Study Limitations Raise Questions about the 

Adequacy and Completeness of the Mobility Capabilities Study and Report, GAO-06-938 

(Washington, D.C.: Sept. 20, 2006). This report includes a list of questions to ask in 

assessing the application of generally accepted research standards. To develop these 

standards, we reviewed research literature and DOD guidance and identified frequently 

occurring, generally accepted research standards that are relevant for defense studies 

that define a quality or sound and complete study. The following were our sources for 

these standards, with the citations for the current versions of these documents: GAO, 

Government Auditing Standards: 2011 Revision, GAO-12-331G (Washington, D.C.: 

December 2011); GAO, Designing Evaluations, GAO-12-208G (Washington, D.C.: 

January 2012); GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.: 

February 2004); RAND Corporation, RAND Standards for High-Quality Research and 

Analysis, CP-413 (5/15) (Santa Monica, CA.: May 2015); Air Force Office of Aerospace 

Studies, Analysts Handbook: On Understanding the Nature of Analysis (January 2000); 

Air Force, Office of Aerospace Studies, Air Force Analysis Handbook, A Guide for 

Performing Analysis Studies: For Analysis of Alternatives or Functional Solution Analysis

(July 2004); Department of Defense, DOD Modeling and Simulation (M&S) Verification, 

Validation, Accreditation (VV&A), Instruction 5000.61 (Washington, D.C.: Dec. 9, 2009); 

Department of Defense, Support for Strategic Analysis, Directive 8260.05 (Washington, 

D.C.: July 7, 2011); and Department of Defense, Implementation of Data Collection, 

Development, and Management for Strategic Analyses, Instruction 8260.2 (Washington, 

D.C.: Jan. 21, 2003). We determined the standards were applicable for evaluating the 

Study.
Page 4 GAO-16-820 Military Health Modernization

data used in their analyses. We did not independently assess the 

reliability of the source data that DOD used in developing the Study. 

Rather, as part of our examination of the Study’s adherence to the 

relevant generally accepted standards for defense research, we asked 

DOD officials what steps they took to assess the reliability of the data 

used in developing their respective sections of the final report, and we 

reviewed supporting documentation provided by the officials to identify 

any known data reliability issues.

To address our third objective, we compared the 2016 Study’s cost 

savings estimates for the changes to 10 military hospitals recommended 

in the Study and the related achievement of the Study’s goals for 

transferring health care to DOD’s own facilities and increasing active-duty 

provider productivity with key practices we derived from cost-estimating, 

budgeting, and related guidance. The practices call for, among other 

things, including all significant costs and assessing the reliability of the 

data used to develop the estimate.7 We discussed the Study’s estimate 

with officials from the Study team within the Defense Health Agency.

To address all of our objectives, we met with representatives from 

relevant organizations across DOD, including from the Office of the 

Assistant Secretary of Defense (Health Affairs), the Defense Health 

Agency, and representatives of the medical commands and agencies of 

the Army, Navy, and Air Force.

We conducted this performance audit from October 2015 to September 

2016 in accordance with generally accepted government auditing 

 7

We reviewed numerous federal guidance documents related to cost estimating, 

accounting standards, economic analysis, and budgeting and identified broad themes that 

can be applied to evaluating cost analyses and estimates, including those of cost savings. 

The guidance documents we reviewed include: GAO, GAO Cost Estimating and 

Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, 

GAO-09-3SP (Washington, D.C.: March 2009); Office of Management and Budget 

Circular No. A-11, Preparation, Submission and Execution of the Budget (August 2011, 

superseded by an August 2012 issuance); Federal Accounting Standards Advisory Board, 

Statement of Federal Financial Accounting Standards 4 (June 2011); Department of 

Defense Instruction 7041.3, Economic Analysis for Decisionmaking (Nov. 7, 1995); and 

Department of Defense Financial Management Regulation 7000.14-R, Volume 4, Chapter 

22, Cost Funding (May 2010). Although each of these documents may not apply to these 

circumstances as a legal matter, we believe that they collectively contain broad themes 

that can be applied to evaluating cost analyses.
Page 5 GAO-16-820 Military Health Modernization

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

The MHS is a complex organization that provides health services to 

almost 10 million beneficiaries across a range of care venues, including 

its own direct care system in hospitals and clinics, through its purchased 

care network of private providers, and on the battlefield in support of 

contingency operations. Responsibility for the delivery of care is shared 

among the Office of the Assistant Secretary of Defense (Health Affairs), 

the military services, and the Defense Health Agency. The Office of the 

Assistant Secretary of Defense (Health Affairs) reports to the Under 

Secretary of Defense for Personnel and Readiness, who in turn reports to 

the Secretary of Defense. The Army and Navy medical commands and 

the Air Force medical agency report through their service chiefs to their 

respective military department Secretaries and then to the Secretary of 

Defense. The Office of the Assistant Secretary of Defense (Health Affairs) 

manages the Defense Health Program appropriation, which funds the 

service medical departments. However, the military treatment facilities, 

including hospitals and clinics, are under the direction and control of the 

services, which maintain the responsibility to staff, train, and equip those 

commands to meet mission requirements. The MHS collaboratively 

develops strategy to meet policy directives and targets. The service 

components and the Defense Health Agency are responsible for 

execution. Figure 1 depicts the MHS organizational structure.

Background

Overview and Structure of 

the MHS
Page 6 GAO-16-820 Military Health Modernization

Figure 1: Military Health System (MHS) Organizational Structure 

The Study was chartered by the Assistant Secretary of Defense (Health 

Affairs) and led by a team including representatives from that office, as 

well as from the Defense Health Agency and from the medical commands 

and agencies of the Army, Navy, and Air Force. Officials were organized 

into four working groups that collectively made up the Study team. Study 

officials told us that the majority of the Study’s work was completed in 

early 2013, but DOD did not finalize the Study until early 2015, and it did 

not submit the Study to the congressional defense committees until 

February 2016. The Study included eight recommendations for further 

study and analysis in a wide range of areas, such as defining medical 

force readiness. The areas addressed in the Study included the following:

• Medical Readiness Personnel Requirements: The Study team sought 

to determine the number of active-duty uniformed medical personnel 

needed to support military operations.

• Health Care Demand: The Study team analyzed the demand for 

health care in geographical medical markets both within DOD’s direct 

care system of hospitals and clinics and from its purchased care 

network of private providers. The Study developed goals for the 

Overview of the Military 

Health System 

Modernization Study’s 

Findings and 

Recommendations
Page 7 GAO-16-820 Military Health Modernization

transfer of health care from the private sector to DOD’s own facilities 

both to increase the efficiency of its spending by making better use of 

active-duty providers and to provide sufficient workload for them to 

maintain their skills.

• Provider Currency Requirements: The Study team sought to develop 

a metric to assess the currency of medical personnel—that is, an 

assessment of the maintenance of clinical skills. Currency, in turn, 

supports the readiness of medical personnel, meaning their ability to 

provide medical support during operations. However, the Study team 

was unable to agree upon a clinical currency measure, and instead 

adopted a productivity metric as a proxy for currency. The productivity 

metric aims to measure active-duty providers’ workload level 

measured in Relative Value Units, a measure of health care 

production. The accompanying goal is for active-duty specialist 

providers to achieve 40 percent of the median productivity of 

specialist private-sector providers as defined by a major medical 

survey.8

• Provider Allocation Model: The Study team developed a model that 

matched uniformed active-duty medical providers with demand for 

health care within geographic medical markets, with the assumption 

that the aforementioned goals for the transfer of health care from the 

purchased network of private providers to DOD’s facilities and 

increasing the productivity of active-duty providers will be achieved. 

This process was intended to demonstrate over- or undersupply of 

providers in certain specialties. The analysis found that for some 

specialties, there was insufficient demand or other constraining 

factors that prevented the matching of all providers to markets. For 

example, based on the Study’s assumptions regarding the transfer of 

health care to DOD facilities and increasing active-duty provider 

productivity, 32 orthopedic surgeons could not be matched to medical 

markets, based on 2013 data.

• Primary-Care Analysis: The Study examined the ratio of enrolled 

beneficiaries to primary-care providers, and found that the MHS was 

close to its goal of 1,100 enrollees per primary-care provider.

• Overseas Hospital Analysis: Due to a variety of cultural, logistical, and 

readiness concerns, the Study recommended no change to overseas 

 8

The productivity goal is based on data from the 2012 Medical Group Management 

Association Survey.
Page 8 GAO-16-820 Military Health Modernization

military hospitals. Officials told us that this was the primary issue of 

contention within DOD due to concerns surrounding the potential 

need for these facilities in support of operations.

The Study also included a facility analysis of 24 small hospitals within the 

MHS to determine their future medical capabilities, such as whether they 

should continue to have inpatient facilities or birthing centers.9 The Study 

recommended changes for 10 of the 24 hospitals, including closing 

inpatient services at 8 of them. The Under Secretary of Defense for 

Personnel and Readiness approved proposed changes for 8 of the 10 

facilities.

The Study estimated annual net cost savings of $366 million in fiscal year 

2012 dollars for the recommended changes to the 10 affected hospitals 

and for achieving its goals for transferring health care to DOD facilities 

and increasing active-duty provider productivity. This estimated cost 

savings included about $132 million of savings associated with reductions 

in civilian Full Time Equivalents (FTE), $333 million of savings associated 

with decreased purchased care costs, and $98 million in increased 

nonlabor costs.10 Prior to the release of DOD’s report to Congress, the 

Navy completed changes to two of the hospitals and retained, though 

reduced, inpatient services at the third. The Army also approved selected 

changes at three of its facilities.

 9

The smallest 10 were selected before the development of the Provider Allocation Model, 

but officials told us that these facilities were unlikely to have any providers allocated to 

them under the model. The next 14 were selected based on the results of the model and 

average daily patient load data.

10All figures are rounded.
Page 9 GAO-16-820 Military Health Modernization

In the Report on Military Health System Modernization (the Study), DOD 

included information that addressed four of the elements listed in section 

713 of the NDAA for Fiscal Year 2015, partially addressed four elements, 

and did not address one element.11 For example, the Study addressed 

the element concerning the extent to which DOD consulted with 

community hospitals to determine their ability to absorb additional 

patients by discussing with local TRICARE experts. Similarly, DOD 

addressed the element concerning the recommendations of the Secretary 

to restructure or realign military medical treatment facilities by including a 

list of recommended changes to 10 such facilities.

However, the Study partially addressed four elements by not addressing 

every item of the relevant element or including incomplete information. 

For example, the Study partially addressed the element concerning the 

impact of recommended changes to military treatment facilities on the 

access of eligible beneficiaries to high-quality health care, and 

satisfaction with such care. While the Study listed the criteria by which 

DOD evaluated the impact of recommended changes on beneficiaries’

access to high-quality health care and satisfaction with such care, it did 

not provide further details on the results of its evaluation. Similarly, DOD 

partially addressed the element that listed 14 items concerning hospitals 

considered under the Study from 2006 to 2012. While the Study provided 

information addressing 11 of these items, it did not provide full information 

for the remaining 3, such as the omission of budgeted funding levels for 

Navy facilities from 2006 through 2009. The Study did not address the 

element concerning the impact that proposed changes to military medical 

treatment facilities would have on timely access by local civilian 

populations to health care services. While the Study included data on 

civilian hospital capabilities, it did not explain what impact, if any, the 

proposed changes would have on access to such facilities. Appendix II 

provides further information on our analysis of the extent to which the 

Study addressed the nine elements listed in Section 713 of the NDAA for 

Fiscal Year 2015.

 11Pub. L. No. 113-291, Section 713.

DOD’s Study 

Addressed Four 

Elements, Partially 

Addressed Four 

Elements, and Did 

Not Address One 

Element in the 2015 

National Defense 

Authorization Act
Page 10 GAO-16-820 Military Health Modernization

Six of the nine elements required a single response, such as an 

explanation or analysis. The remaining three elements listed multiple 

items, including two elements that required information concerning all of 

the hospitals considered under the Study. Although DOD assessed 24 

hospitals in its Study, it chose to include information in these two 

elements that pertained only to the 8 hospitals for which the Secretary of 

Defense approved the recommended changes. Neither the Study nor the 

appendix provided a full explanation as to why not every aspect of the 

required items was included. The Study appendix noted in one instance, 

for the item concerning mental-health availability at training centers, that 

there were limited data available.

According to senior Study team officials, DOD compiled the data required 

by Section 713 after the Study was finalized in May 2015. Further, 

officials stated that, in general, the compilation of these data did not 

inform the Study’s analysis, findings, or recommendations, because DOD 

had initiated the study on its own initiative prior to the enactment of the 

NDAA for Fiscal Year 2015.12 Although Study team officials stated that 

they independently analyzed some of the same data elements as those 

listed in the NDAA, they did not examine all such elements. However, for 

the data elements they did examine, they may not have examined the 

exact attributes specified in the NDAA.

Our analysis of the Study found that it followed some, but not all, of the 

relevant generally accepted research standards for research design and 

execution.13 Such standards contribute to a sound and complete study. 

According to these standards, a study’s design should include, for 

example, establishing the objectives, scope, and methodology, and 

identifying study assumptions. Successful execution involves ensuring 

that the methodology is carried out as planned and ensuring that data 

used in the study are sufficiently valid and reliable for the study’s 

purposes. (See app. I for further information on the generally accepted 

research standards that we used to conduct our assessment.) 

Specifically, we found the following:

 12Pub. L. No. 113-291, § 713.

13GAO-06-938.

DOD’s Study Did Not 

Consistently Follow 

Relevant Generally 

Accepted Research 

Standards
Page 11 GAO-16-820 Military Health Modernization

• The Study’s objectives, scope, and methodology were generally 

consistent with relevant accepted research standards.

• The Study did not sufficiently mitigate limitations concerning its 

analysis of the required number of active-duty medical and civilian 

medical personnel.

• The Study included a limited assessment of clinical currency—the 

requirements necessary to maintain the skills of active-duty medical 

providers.

• The Study established goals for the transfer of health care from 

DOD’s purchased care network into its own network of hospitals and 

clinics and for increasing the productivity of active-duty medical 

providers without a strategy explaining how these goals would be 

achieved.

• The Study’s provider allocation model did not reflect differences 

among the military services.

• The Study did not describe steps taken to ensure the reliability of the 

data supporting its analyses and recommendations.

In general, Study officials, including those from the various working 

groups, stated that they were limited in their analysis by the time available 

to them. Study officials told us that the majority of the Study’s work was 

completed in early 2013, but DOD did not finalize the Study until early 

2015, and did not submit it to the congressional defense committees until 

February 2016.

The Study’s recommendations for future work and analysis position DOD 

to continue important efforts to modernize the MHS. However, without 

consistently following generally accepted research standards, DOD’s 

work on the Study’s recommendations may not be fully sound and 

complete. By addressing the areas that we identified in our review, DOD 

can strengthen its future efforts.
Page 12 GAO-16-820 Military Health Modernization

The Study’s objectives, scope, and methodology were generally 

consistent with relevant accepted research standards for the design of a 

study. Generally accepted research standards state that objectives 

outline what a study is intended to accomplish; a study’s scope defines 

the subject matter that will be assessed and reported on, and is directly 

tied to the objectives; and a study’s methodology should be consistent 

with and address the objectives.14

The Study had five objectives that defined the subject matter to be 

addressed, and all of these related to DOD’s high-level goals for the 

Study of ensuring the readiness of the medical force and creating a more 

efficient MHS. The Study’s objectives generally corresponded to the goals 

in the Study’s Charter document. However, two goals stated in the

Study’s Charter were not addressed in these objectives or in the Study’s 

execution. Specifically, the Study did not address the goals to finalize 

development of an updated joint casualty model, which would provide a 

cross-service assessment of the need for medical support in support of 

operations, and to develop recommended civilian medical force levels. 

(The efforts by the Study team to address the Study’s objectives and 

Study’s Charter goals are discussed later in this report.) Although the 

Study did not address all of its Charter’s goals, it provided new 

information in some areas, such as currency of active-duty providers, 

which DOD can use to develop future analysis.

The Study’s scope and methodology were linked with the objectives, 

consistent with generally accepted research standards. The Study’s 

scope included all medical personnel and facilities that deliver direct 

patient care in the Defense Health Program–funded treatment facilities. 

The Study’s methodology involved various analyses by the working 

groups that were structured around each of the five objectives. The 

Study’s work was carried out by corresponding working groups, overseen 

by an overarching group. The Study also provides a brief discussion of 

“risks and mitigation strategies” comprising a selected list of six changes 

in the environment or underlying assumptions that could affect the 

Study’s recommendations and an accompanying potential alternative 

mitigation strategy to address each change.

 14GAO-06-938.

Objectives, Scope, and 

Methodology Were 

Generally Consistent with 

Relevant Accepted 

Research Standards
Page 13 GAO-16-820 Military Health Modernization

The Study presented data on the size and composition of the wartime 

active-duty medical force, also known as the “operational force.”

However, the data were largely drawn from service models with known 

problems and not based on new analysis conducted by the Study team. 

To estimate the size of the operational force, the Study presented the 

sum total of the services’ individual estimates of their needs. These 

estimates are produced by service-specific models and planning tools 

that examine a common set of defense planning scenarios. However, the 

Study highlighted differences in approach between the services. For 

example, the Army chose to use its own estimating tool because the Joint 

Medical Planning Tool was not accredited at the time of the Study. In 

contrast, the Navy chose to use an early, unaccredited version of the 

Joint Medical Planning Tool. Similarly, while the Army used the Joint 

Integrated Contingency Model to develop its casualty estimates, which 

are an input into the various planning tools, the Navy used four separate 

models, including two prototypes. The Air Force does not use casualty 

estimates, but rather an estimate of the number of likely evacuees from 

theater. The Study noted that a separate independent analysis conducted 

by the Joint Staff correlated with the services’ estimates for the 

operational force. In addition, Study team officials stated that the services’

processes for developing their “generating” and “institutional” forces, 

which include personnel responsible for issues such as graduate medical 

education and headquarters staff functions, were not shared with the 

Study team. Previous reviews of the necessary size of the medical force 

have highlighted the differences between analyses based on servicegenerated estimates and analyses based on DOD-wide estimates. 

Specifically, during the 1990s, two analyses based on DOD-wide 

estimates found the services had excess medical personnel relative to 

their needs, while one analysis based on service-generated estimates 

found the services needed nearly all of their medical personnel to meet 

their needs.15

Generally accepted research standards state that limitations and 

constraints, which are generally found in studies, should be identified and 

reasonably mitigated.16 While the Study clearly outlined the limitations of 

 15These studies arose from requirements under the National Defense Authorization Act 

(NDAA) for Fiscal Year 1992.

16GAO-06-938.

The Study Did Not Fully 

Mitigate Limitations 

Concerning Its Analysis of 

the Required Number of 

Active-Duty and Civilian 

Medical Personnel
Page 14 GAO-16-820 Military Health Modernization

its service-oriented approach, the Study did not fully mitigate them, such 

as by explaining how these limitations affected the results of the analysis. 

Officials with the working group that addressed readiness requirements 

stated that their methodology was limited by the time available for 

analysis. Specifically, the officials stated that they knew they would be 

unable, in the time allowed, to conduct a full, independent analysis of the 

number of active-duty medical personnel necessary to support 

contingency operations, and that they therefore relied upon estimates 

from the military services. Officials stated that it would be feasible to 

conduct an independent analysis of the required number of active-duty 

medical personnel if there were greater information sharing from the 

services and more time to complete such an analysis. While we recognize 

the time constraints faced by the Study team, addressing these issues 

would provide more meaningful results as DOD considers further 

changes to the MHS. Absent such efforts to address these limitations, 

DOD will not have a full assessment of its medical workforce needs.

In addition, the Study’s Charter included a goal to determine civilian 

personnel levels, but the Study did not include such an assessment in its 

scope. Civilians represent a significant proportion of the MHS workforce. 

For example, in 2015, civilians represented 61 percent of the Army’s 

medical positions. Civilian personnel levels also have a direct effect on 

active-duty personnel levels, and DOD policy requires the services and 

defense agencies to determine the appropriate mix of manpower (military 

and civilian) and private-sector support.17 DOD’s Medical Readiness 

Review, an outgrowth of the 2006 Quadrennial Defense Review, found 

that efficiencies could be made by converting military billets to civilian 

positions, although a prohibition on such conversions was later put in 

place. Without addressing the role of civilians in the MHS, the Study’s 

analysis does not present a complete picture of DOD’s needs.

The Study included a limited assessment of clinical currency that did not 

sufficiently identify or mitigate several limitations and constraints. As 

noted earlier, the Study team was unable to agree upon a measure of the 

clinical currency of active-duty medical personnel—that is, an assessment 

 17Department of Defense Instruction 1100.22, Policy and Procedures for Determining 

Workforce Mix (Apr. 12, 2010).

The Study Included a 

Limited Assessment of 

Clinical Currency
Page 15 GAO-16-820 Military Health Modernization

of the maintenance of skills. Currency, in turn, supports the readiness of 

medical personnel, meaning their ability to provide medical support during 

operations. The Study stated that the Study team attempted to identify an 

industry standard for clinical currency, but none existed. In addition, a 

Study official stated that the working group had a limited time frame to 

complete its work. The Study team instead adopted a productivity metric 

as a proxy for currency, along with an accompanying goal for active-duty 

specialist providers to achieve 40 percent of the median productivity of 

private-sector specialist providers as defined by a major medical survey.

However, some Study team officials expressed doubts concerning the 

accuracy of provider productivity data used to develop the proxy measure 

for clinical currency. Specifically, officials stated that they believe that the 

MHS as a whole does not accurately record Relative Value Units,18

meaning that the MHS does not accurately record providers’ workload. 

Further, a senior Study team official stated the military services and 

individual military treatment facilities differ in how they record this 

information, and, due to these differences, the Study team adjusted the 

Relative Value Unit figures for a number of hospitals, sometimes with a 

margin of error of up to 50 percent. Generally accepted research 

standards state that limitations and constraints should be identified and 

reasonably mitigated. However, the limitations in the accuracy of 

information concerning providers’ workload were not identified or 

mitigated in the Study. Having accurate workload information is important 

to establishing a sound standard for maintaining the clinical skills of 

medical providers.

Further, in comparing the productivity of active-duty medical providers 

with civilian medical providers, the Study reduced the amount of time 

active-duty providers are available to practice medicine because of the 

“readiness cost” of time that they must devote to military-specific duties 

and requirements, such as physical training, military training, and federal 

holidays. The Study team’s approach relied on subject-matter experts to 

estimate the time active-duty providers had available to practice 

medicine, and a Study official said there were no reliable data concerning 

the average number of clinical hours worked by active-duty providers. 

 18Relative Value Units are a metric of health-care output that is calculated based on the 

type and complexity of an episode of provision of care.
Page 16 GAO-16-820 Military Health Modernization

Other approaches, such as time studies of active-duty providers, are 

more rigorous, but the Study official stated that the working group’s 

approach was limited by time constraints.

The Study contained a recommendation to further study the costs of 

medical force readiness, and DOD is currently working to establish a 

metric for medical force readiness. However, without the benefit of more

rigorous analysis concerning active-duty providers’ workload and the time 

they must devote to military-specific duties and responsibilities, DOD will 

not have a sound and accurate standard for maintaining the clinical skills 

of medical providers.

The Study, in developing its findings and recommendations, assumed 

that DOD would be successful in its goals for transferring health care from 

patients being treated in DOD’s purchased private-sector network into its 

own military treatment facilities, as well as in achieving related goals for 

improving provider productivity. The Study recommended the 

establishment of associated goals, and DOD adopted such goals in 

March 2014. However, this approach was not consistent with the 

generally accepted research standard that assumptions should be 

reasonable. Specifically, the Study’s assumptions regarding its goals for 

transfer of health care to DOD’s facilities and increasing active-duty 

provider productivity are unsupported because DOD has not developed 

an accompanying strategy describing how these goals can be achieved.

Our prior work has identified six selected leading practices of effective 

federal strategic planning, which we derived in part from the Government 

Performance and Results Act (GPRA), as updated by the GPRA 

Modernization Act of 2010, associated guidance, and our prior work.19

Specifically, these leading practices are to: (1) define the mission and 

goals, (2) define strategies that address management challenges and 

identify resources needed to achieve goals, (3) ensure leadership 

 19For example, see GAO, Executive Guide: Effectively Implementing the Government 

Performance and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1, 1996); Tax 

Administration: IRS Needs to Further Refine Its Tax Filing Season Performance 

Measures, GAO-03-143 (Washington, D.C.: Nov. 22, 2002); and Managing for Results: 

Strengthening Regulatory Agencies’ Performance Management Practices, 

GAO/GGD-00-10 (Washington, D.C.: Oct. 28, 1999).

The Study Established 

Goals for Transferring 

Health Care into Its Own 

Network of Hospitals and 

Clinics and Increasing the 

Productivity of Providers 

without a Strategy 

Explaining How These 

Goals Would Be Achieved
Page 17 GAO-16-820 Military Health Modernization

involvement and accountability, (4) involve stakeholders, (5) coordinate 

with other federal agencies, and (6) develop and use performance 

measures. Study team officials told us that, notwithstanding the absence 

of a strategy, transfer of health care to DOD facilities is a priority and that 

such goals were modest and achievable. However, the Study set out 

goals for transferring health care into its own facilities for large groups of 

beneficiaries, such as TRICARE for Life beneficiaries, for whom, unlike 

active-duty military and their families, the MHS has few tools to 

encourage their use of military treatment facilities.

In addition, transfer of health care into DOD facilities has been an MHS 

goal for several years, but limited progress has been made. For example, 

the Congressional Budget Office reported in 201420 that most of the 

growth in outpatient utilization of health care services within the MHS has 

occurred within the purchased care system, while military hospitals and 

clinics have maintained a relatively constant workload since 2005. 

Similarly, it is uncertain that DOD can achieve its related productivity 

goals without an accurate means for measuring providers’ workload, as 

discussed earlier. Without a strategy that incorporates leading practices, it 

remains unclear whether or how DOD can achieve its goals to transfer 

health care from the purchased care network into its own network of 

facilities.

The Study included a provider allocation model that identified the medical 

specialties needed with anticipated health care demand within specific 

geographic medical markets. The model’s results highlighted specialties 

for which the MHS has too many medical specialists relative to 

anticipated market demand for various health care services and other 

specialties for which the MHS has too few physicians to accommodate 

market demand. Generally accepted research standards state that 

models should, among other things, represent a reasonably complete 

range of conditions and be appropriate for the intended purpose.21

However, an important aspect of the provider model did not reflect the 

standards in that it unreasonably assumed that medical providers from 

 20Congressional Budget Office, Approaches to Reducing Federal Spending on Military 

Health Care (January 2014). 

21GAO-06-938.

The Study’s Provider 

Allocation Model Did Not 

Reflect Differences among 

the Military Services
Page 18 GAO-16-820 Military Health Modernization

the different services were interchangeable. Specifically, the Study’s 

model for allocation of specialist physicians based on anticipated health 

care demand in geographic markets did not account for the militaryservice ownership of the relevant hospitals and the physicians’ military 

service. Instead, it matched the number of physicians across all services 

within a given specialty, such as orthopedic surgery, to anticipated health 

care demand in geographic markets, regardless of the service ownership 

of the medical treatment facility in that market. While such arrangements 

do exist in “multiservice markets,” such as the National Capital Region 

and San Antonio, they are less common elsewhere. The model could 

potentially provide results that do not reflect MHS business practices in 

the real world. For example, it could suggest that there are sufficient 

physicians to provide a given type of specialty care to a particular 

geographic medical market, when in reality there are not sufficient 

physicians from that medical market’s primary military service to assign. 

Currently, active-duty providers generally practice within their own 

services’ hospitals in a team context unique to their military service. The 

Study emphasized the view that it is important for physicians to practice 

medicine within this team context of other specialists and nurses from the 

same service, which can be seen within the services’ different 

configurations of medical personnel in a deployed setting.

CNA independently assessed one part of DOD’s model. However, its 

assessment included only the mechanics of the model and not an 

examination of the supporting data used in the model. The Study stated 

that the Study team relied on military-service representatives to verify 

these data. In addition, CNA noted that alternative approaches could yield 

different results depending on assumptions concerning transfer of health 

care to DOD facilities and increased active-duty provider productivity. The 

Study asserted that its modeling results were reasonable given the time 

constraints of the Study. The Study recommended further work to refine 

and formalize the model and use it to inform decisions concerning the 

delivery of health care services. However, unless DOD addresses 

differences between the services and the effects of the model’s practical 

application, the model’s usefulness will be limited and could lead to 

impractical decisions.
Page 19 GAO-16-820 Military Health Modernization

The Study and the data appendix did not describe steps that the Study 

team took to ensure that the data used to support the Study and data 

appendix were sufficiently reliable for the Study’s purposes. Generally 

accepted research standards state that the data used in a study should 

be valid for the intended research purpose; in other words it represents 

what it purports to represent.22 In addition, documenting a study’s 

analytical and decision-making process helps to ensure the accuracy, 

completeness, and credibility of study results. However, we found that the 

study presented limited information on its efforts to assess data reliability. 

For example, the Study’s data appendix did not document the source of 

its data in several instances. Specifically, the data source is provided for 9 

of 15 data metrics, but not for the other 6 metrics.

Because the Study presented limited information on the Study team’s 

efforts to assess the reliability of the data used, we provided each working 

group’s lead official with a standard set of questions concerning their 

efforts to ensure data reliability. Specifically, we asked the working group 

leads which data sources they used in their respective analyses. Their 

responses indicated that they used five databases and one data set that 

were not mentioned in the main body of the Study or the data appendix. 

We questioned the working group leads concerning steps taken to ensure 

data reliability, such as whether they consulted with the source or owner 

of the data to determine what methods were used to ensure reliability or 

whether they took other steps to ensure reliability, such as conducting 

their own assessment or reliance on existing assessments. Working 

group officials stated that they generally relied on the reliability processes 

of the owners of the data to ensure the data provided were reliable. For 

example, two of the working groups stated that they used databases that 

are commonly used within the MHS, and that these databases are subject 

to oversight reviews and periodic validity checks. An official from one 

working group stated that the group also conducted reviews of relevant 

data to check for completeness of records related to inpatient care and 

found missing data. The working group developed adjustment factors and 

applied them by market and provider specialty to impute missing data 

where possible. An official from another working group stated that the 

group used data provided by contractors, and that DOD’s subject-matter 

experts routinely discuss report content with the contractors as a way to 

 22GAO-06-938.

The Study Did Not 

Describe Steps Taken to 

Ensure the Reliability of 

the Data Supporting Its 

Analyses and 

Recommendations
Page 20 GAO-16-820 Military Health Modernization

verify the validity and reliability of the data provided as part of the 

performance assessment of the contract. In addition, we interviewed 

service officials regarding the accuracy and reliability of the data used in 

the Study, and officials from the Navy disputed the accuracy of the 

Study’s reported physician productivity data. Army officials stated that 

they had some differences in data interpretation based on their own 

analysis, which led them to some different recommendations.

When asked if there were any limitations on the use of specific data 

sources, working group officials reported to us some limitations that were 

not specifically described in the main body of the Study or data appendix. 

For example, one working group official mentioned that military treatment 

facility bed counts and ambulatory surgery counts were not consistently 

reported, though officials stated that these data did not appear to have a 

significant effect on the group’s findings. Another limitation concerned the 

consistency of data from the Medical Expense and Performance 

Reporting System affecting the assessment of a primary-care provider’s 

availability. An official noted that as a result of this limitation, the final 

analysis of primary-care capacity was curtailed and specific 

recommendations were limited.

A systematic review of over 300 supporting work papers provided to us by 

Study officials found multiple examples of known data or analysis 

limitations that, if the Study was adhering to generally accepted research 

standards, would have been conveyed in the Study. For example, one of 

the work papers noted data reliability issues with certain overseas-related 

data which were not noted in the report. One work paper noted some 

limitations in a data source used in the study that were not discussed in 

the report, and another work paper mentioned having to manually adjust 

some data metrics, which was not noted in the Study. Conveying data or 

analysis limitations is important so that decision makers relying on the 

Study can fully understand the quality and soundness of the research. 

Without sufficiently demonstrating the accuracy, reliability, and limitations 

of the data upon which DOD makes policy changes, decision makers do 

not have assurance that the Study’s findings and recommendations are 

based on sound information.
Page 21 GAO-16-820 Military Health Modernization

DOD’s estimated cost savings for the recommended changes to 10 small 

hospitals and the related achievement of goals related to transfer of 

health care to DOD facilities and increased active-duty provider 

productivity included a net estimate of savings and costs, but did not fully 

reflect key practices for cost and cost savings estimates. For example, 

DOD included estimated savings from three categories—reductions in 

civilian personnel, net decreases in purchased care, and net increases in 

nonlabor costs—and listed general sources of potential savings. 

However, the Study did not fully reflect key practices we derived from 

cost-estimating, budgeting, and related guidance. These key practices 

emphasize the importance of an appropriate level of detailed 

documentation, the inclusion of all significant costs, and an assessment 

of the reliability of the data used to develop the estimate, among others.23

First, the Study did not provide an appropriate level of detail concerning 

the calculation of the estimated cost savings or the assumptions upon 

which they were based. For example, the Study’s cost savings estimate 

did not include details concerning the number of military personnel who 

would be moved from the 10 affected facilities to achieve the savings 

associated from transfer of health care to DOD facilities and increased 

active-duty provider productivity. In addition, of the total estimated 

savings, $333 million was associated with changes that were based on an 

assumption concerning DOD’s ability to achieve its goals related to 

transfer of health care to DOD facilities and increased active-duty 

provider productivity without stating a strategy to reach those targets. 

Absent a clear and likely achievable strategy, the predicted cost savings 

associated with transfer of health care to DOD facilities and increasing 

active-duty provider productivity are questionable.

 23We reviewed numerous federal guidance documents related to cost estimating, 

accounting standards, economic analysis, and budgeting and identified broad themes that 

we believe can be applied to evaluating cost analyses and estimates, including those of 

cost savings. The guidance documents we reviewed include GAO-09-3SP; Office of 

Management and Budget Circular No. A-11, Preparation, Submission and Execution of 

the Budget; Federal Accounting Standards Advisory Board, Statement of Federal 

Financial Accounting Standards 4; Department of Defense Instruction 7041.3, Economic 

Analysis for Decisionmaking (Nov. 7, 1995); and Department of Defense Financial 

Management Regulation 7000.14-R, Volume 4, Chapter 22, Cost Funding. Although each 

of these documents may not apply to these circumstances as a legal matter, we believe 

that they collectively contain broad themes that can be applied to evaluating cost 

analyses.

DOD’s Estimate of 

Cost Savings from 

Changes to Small 

Hospitals Did Not 

Fully Reflect Key 

Practices
Page 22 GAO-16-820 Military Health Modernization

Second, the Study’s predicted net savings estimate did not include all 

potentially significant costs. Specifically, its cost savings estimate did not 

address the effect of the following costs:

• Costs associated with a reduction-in-force of civilian FTEs: The Study 

estimated savings of $132 million associated with the elimination of 

1,324 civilian FTEs across the MHS due to the aforementioned 

assumption of greater productivity of active-duty providers. However, 

we have previously reported that DOD’s efforts to reduce civilian 

FTEs do not always result in planned decreased costs of a 

proportionate magnitude to the decrease in FTEs. For example, in 

2016, we found that across DOD, from fiscal years 2012 through 

2016, civilian FTEs declined by 3.3 percent, but civilian personnel 

costs declined by only 0.9 percent, adjusted for inflation.24

• Onetime costs to realign active-duty providers to other hospitals: The 

Study recommended changes that would realign active-duty providers 

to other facilities. However, movement of these providers, the 

associated active-duty medical professionals with whom they work in 

a team setting, and their families will incur costs such as permanent 

change of station expenditures. Study officials told us that they did not 

include these costs because permanent change of station costs vary 

widely and final decisions concerning where personnel would be 

relocated rest with the military services. We previously reported on the 

costs of permanent change of station moves and found that these 

moves, on average, cost about $6,700 per servicemember in 2014.25

• Construction costs, if applicable, at facilities losing or gaining activeduty providers: The Study recommended that a number of inpatient 

facilities be closed or converted to ambulatory surgery centers, but 

our analysis found that the Study did not identify estimated costs 

associated with these changes. Construction costs also may be 

applicable at facilities gaining active-duty providers, but Study officials 

stated that execution of such changes would be the responsibility of 

 24GAO, Civilian and Contractor Workforces: Complete Information Needed to Assess 

DOD’s Progress for Reductions and Associated Savings, GAO-16-172 (Washington, D.C.: 

Dec. 23, 2015). 

25GAO, Military Compensation: DOD Needs More Complete and Consistent Data to 

Assess the Costs and Policies of Relocating Personnel, GAO-15-713 (Washington, D.C.: 

Sept. 9, 2015).
Page 23 GAO-16-820 Military Health Modernization

the services. However, without specifically addressing these costs or 

justifying their exclusion, the Study’s cost savings estimate leaves 

open the question of whether potential savings could be reduced by 

such costs.

Third, DOD’s savings estimate did not describe the steps taken by the 

Study team to assess the reliability of cost data used to develop the cost 

savings estimate. As discussed earlier in this report, the Study did not cite 

the data sources from which the cost estimates were drawn, and did not 

discuss what limitations, if any, such data might have had for the 

purposes to which they were applied.

As a result of these weaknesses, DOD may not have accurately 

estimated the net savings that might result from its proposed hospital 

changes. As DOD considers additional changes to its network of military 

treatment facilities, decision makers do not have a full and accurate 

assessment of all relevant costs and savings. Without a full and accurate 

assessment, DOD is not in a position to determine whether its efforts to 

respond to the Study’s recommendations will result in savings or, 

alternatively, in additional costs.

In initiating the Study, DOD began the process of examining difficult and 

complex problems affecting the MHS. The Study’s recommendations for 

further analysis and work in a number of areas position DOD, over time, 

to explore these areas and take actions to improve the effectiveness and 

efficiency of the MHS. However, our analysis of the Study highlighted 

shortcomings that merit review and mitigation before DOD takes definitive 

action on these recommendations. Specifically, without addressing the 

issues discussed in our report, DOD’s work to address the Study’s 

recommendations will continue to reflect the various unsupported 

assumptions, unmitigated limitations and constraints, and questions 

concerning the reliability of data used to develop the Study’s 

recommendations that we have highlighted. Similarly, as DOD considers 

further changes to its network of facilities and the medical services they 

offer, the analysis supporting these decisions will be incomplete without a 

full and accurate picture of the true cost of such changes.

To fully assess the size and composition of the medical force, we 

recommend that the Secretary of Defense direct the Assistant Secretary 

of Defense (Health Affairs) to conduct a new analysis of the required 

Conclusions

Recommendations for 

Executive Action
Page 24 GAO-16-820 Military Health Modernization

number of active-duty and civilian medical personnel that mitigates known 

limitations.

To strengthen ongoing efforts to analyze the costs of medical force 

readiness and establish clinical currency standards, we recommend that 

the Secretary of Defense direct the Assistant Secretary of Defense 

(Health Affairs) to take steps to identify and mitigate limitations regarding 

the standard for maintaining providers’ clinical skills, including improving 

the accuracy of information concerning providers’ workload and 

conducting an analytically rigorous calculation of active-duty providers’

time devoted to military-specific responsibilities.

To help achieve DOD’s goals for transferring health care into its own 

facilities and increasing the productivity of active-duty medical providers, 

we recommend that the Secretary of Defense direct the Assistant 

Secretary of Defense (Health Affairs) to develop a strategy for achieving 

these goals that reflects the leading practices of effective federal strategic 

planning.

To strengthen ongoing efforts within DOD to address the Study’s 

recommendations to use the provider model outputs to inform execution 

of health care delivery and to refine the model for future use, we 

recommend that the Secretary of Defense direct the Assistant Secretary 

of Defense (Health Affairs) to modify DOD’s model to reflect the military 

service of the physicians and military treatment facilities included in the 

model.

To strengthen any future assessments of additional changes to DOD’s 

network of military treatment facilities, we recommend that the Secretary 

of Defense direct the Assistant Secretary of Defense (Health Affairs) to 

take the following two actions:

• describe steps taken to assess the reliability of data supporting the 

assessment, including, at a minimum, the sources of data, data 

limitations, and efforts to test data reliability; and

• include in any accompanying cost estimates an appropriate level of 

detail, all significant costs, and an assessment of the reliability of the 

data supporting the cost estimate.

In written comments on a draft of this report, DOD concurred with our six 

recommendations to improve DOD’s analysis of the size, readiness, and Agency Comments
Page 25 GAO-16-820 Military Health Modernization

efficiency of the medical force. DOD’s comments are reprinted in 

appendix III.

We are sending copies of this report to the appropriate congressional 

committees, the Secretary of Defense, the Deputy Secretary of Defense, 

the Under Secretary of Defense for Personnel and Readiness, the 

Assistant Secretary of Defense (Health Affairs), the Defense Health 

Agency Director, the Surgeon General of the Air Force, the Surgeon 

General of the Army, and the Surgeon General of the Navy. In addition,

the report is available at no charge on the GAO website at 

http://www.gao.gov.

If you or your staff have any questions regarding this report, please 

contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made key contributions to this 

report are listed in appendix IV.

Brenda S. Farrell

Director

Defense Capabilities and Management
Page 26 GAO-16-820 Military Health Modernization

List of Committees

The Honorable John S. McCain

Chairman

The Honorable Jack Reed

Ranking Member

Committee on Armed Services

United States Senate

The Honorable Thad Cochran

Chairman

The Honorable Richard J. Durbin

Ranking Member

Subcommittee on Defense

Committee on Appropriations

United States Senate

The Honorable Mac Thornberry

Chairman

The Honorable Adam Smith

Ranking Member

Committee on Armed Services

House of Representatives

The Honorable Rodney Frelinghuysen

Chairman

The Honorable Peter J. Visclosky

Ranking Member

Subcommittee on Defense

Committee on Appropriations

House of Representatives
Appendix I: Selection of Generally Accepted 

Research Standards Relevant for the Military 

Health System Modernization Study

Page 27 GAO-16-820 Military Health Modernization

To guide our review of the Department of Defense’s (DOD) Report on 

Military Health System Modernization (the Study), we identified relevant 

generally accepted research standards for the design, execution, and 

presentation of findings that define a sound and complete study. We 

developed these standards by reviewing research literature and DOD 

guidance and identifying frequently occurring, generally accepted 

research standards that are relevant for defense studies. We initially used 

these standards in a prior GAO review of a DOD study on mobility 

capabilities.1 That report includes a list of 36 questions to ask in 

assessing the application of generally accepted research standards. To 

develop these standards, we reviewed research literature and DOD 

guidance and identified frequently occurring, generally accepted research 

standards that are relevant for defense studies that define a high-quality 

or methodologically sound and complete study. The following were our 

sources for these standards:

• GAO, Government Auditing Standards: 2011 Revision, 

GAO-12-331G (Washington, D.C.: December 2011);

• GAO, Designing Evaluations, GAO-12-208G (Washington, D.C.: 

January 2012); 

• GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.: 

February 2004); 

• RAND Corporation, RAND Standards for High-Quality Research 

and Analysis, CP-413 (5/15) (Santa Monica, CA.: May 2015); 

• Air Force Office of Aerospace Studies, Analysts Handbook: On 

Understanding the Nature of Analysis (January 2000); 

• Air Force, Office of Aerospace Studies, Air Force Analysis 

Handbook, A Guide for Performing Analysis Studies: For Analysis 

of Alternatives or Functional Solution Analysis (July 2004); 

 1

GAO, Defense Transportation: Study Limitations Raise Questions about the Adequacy 

and Completeness of the Mobility Capabilities Study and Report, GAO-06-938 

(Washington, D.C.: Sept. 20, 2006). That report includes a list of 36 questions to ask in 

assessing the application of generally accepted research standards.

Appendix I: Selection of Generally Accepted 

Research Standards Relevant for the Military 

Health System Modernization Study
Appendix I: Selection of Generally Accepted 

Research Standards Relevant for the Military 

Health System Modernization Study

Page 28 GAO-16-820 Military Health Modernization

• Department of Defense, DOD Modeling and Simulation (M&S) 

Verification, Validation, Accreditation (VV&A), Instruction 5000.61

(Washington, D.C.: Dec. 9, 2009); 

• Department of Defense, Support for Strategic Analysis, Directive 

8260.05 (Washington, D.C.: July 7, 2011); 

• Department of Defense, Implementation of Data Collection, 

Development, and Management for Strategic Analyses, Instruction 

8260.2 (Washington, D.C.: Jan. 21, 2003).

We determined that these standards are still current and relevant for the 

purposes of this report.

We confined our assessment to the standards related to design (11 of 14 

standards) and execution (10 of 15 standards) because the National 

Defense Authorization Act for Fiscal Year 2015 mandated we assess the 

methodology and data used in the Study. Consequently, we did not 

assess DOD’s report against the generally accepted research standards 

for presentation of results (7 standards). Of the applicable standards 

related to design and execution, we combined and slightly revised the 

standards to match the particular analyses used in the study. Some of the 

design and execution standards were not relevant to our assessment. For 

example, the standards concerning analysis of scenarios and threats are 

specific to certain types of defense studies and did not relate to the study 

that was the subject of our review. We compared DOD’s approach to the 

Study with these selected generally accepted research standards. 

Accordingly, we technically reviewed the content of DOD’s report against 

the following 18 standards that were relevant to our review:

1. Design:

a. Objectives, Scope and Methodology: Objectives outline what a 

study is intended to accomplish. A study’s scope defines the 

subject matter that will be assessed and reported on, and the 

scope is directly tied to the audit objectives. A study’s 

methodology should be consistent with and address a study’s 

objectives.

• Is the study’s design clear?

• Is the study’s objective clearly stated?

• Is the study’s scope clearly defined?
Appendix I: Selection of Generally Accepted 

Research Standards Relevant for the Military 

Health System Modernization Study

Page 29 GAO-16-820 Military Health Modernization

• Do the study scope, methodology, and objectives fully 

address the study charter and associated guidance?

• Was the study plan followed?

• Were deviations from the study plan explained and 

documented?

b. Assumptions: Assumptions should be reasonable, consistent, 

appropriately varied, and explicitly identified in order to understand 

the context of the study’s results. Assumptions define the 

parameters for what will be included and how the study is 

executed.

• Are the assumptions explicitly identified?

• Are the assumptions reasonable and consistent?

• Are the assumptions varied to allow for sensitivity analyses?

c. Limitations and Constraints: Limitations and constraints, which are 

generally found in studies, should be identified and reasonably 

mitigated. Research standards do not call for the elimination of all 

limitations, but do call for them to be identified, assessed, 

mitigated, and explained.

• Are the limitations explicitly identified?

• Are the major constraints identified and discussed?

2. Execution:

a. Models: Models should be reasonable ones to consider, represent 

a reasonably complete range of conditions, as well as be 

appropriate for the intended purpose.

• Are the scenarios that were modeled reasonable ones to 

consider?

• Do the scenarios represent a reasonably complete range of 

conditions?

• Were the models used to support the analyses appropriate for 

their intended purpose? Have they been described and 

documented adequately?

• Were any modeling limitations identified, explained, and 

justified?
Appendix I: Selection of Generally Accepted 

Research Standards Relevant for the Military 

Health System Modernization Study

Page 30 GAO-16-820 Military Health Modernization

b. Data reliability: The data used in a study should be valid for the 

intended research purpose; in other words, they represent what 

they purport to represent. Documenting a study’s analytical and 

decision-making process helps to ensure the accuracy, 

completeness, and credibility of study results.

• Were the data used valid for the study’s purposes?

• Were the data used sufficiently reliable for the study’s 

purposes?

• Were any data limitations identified and was the effect of the 

limitations adequately explained?
Appendix II: Comparison of DOD’s Military 

Health System Modernization Study with 

Requirements Listed in 2015 National Defense 

Authorization Act

Page 31 GAO-16-820 Military Health Modernization

Table 1 summarizes our assessment of the extent to which the 

Department of Defense (DOD) in its Report on Military Health System 

Modernization (the Study) addressed each of the required nine elements 

listed in Section 713 of the National Defense Authorization Act (NDAA) for 

Fiscal Year 2015. Some of the elements required that multiple items be 

addressed. We considered an element “addressed” if it included all of the 

items listed in the NDAA; “partially addressed” if it included some, but not 

all, of the items; and “not addressed” if it did not include any of the items.

Table 1: Comparison of DOD’s Military Health System Modernization Study with Elements Listed in Section 713 of the NDAA 

for Fiscal Year 2015 

National Defense Authorization Act (NDAA) elements and items Our assessment

(A) During the period from 2006 to 2012, for each military medical treatment 

facility considered under the modernization study—

Partially Addressed

(i) the average daily inpatient census Addressed

(ii) the average inpatient capacity Partially addressed: Only two years of data included.

(iii) the top five inpatient admission diagnoses Addressed

(iv) each medical specialty available Addressed

(v) the average daily percent of staffing available for each medical 

specialty

Addressed

(vi) the beneficiary population within the catchment area Addressed

(vii) the budgeted funding level Partially addressed: Did not provide budgeting 

funding level details for the years 2006 through 2009 

for Navy hospitals. Data were only included for three 

Army Community Hospitals. No explanation is 

provided for this omission.

(viii) whether the facility has a helipad capable of receiving medical 

evacuation airlift patients arriving on the primary evacuation aircraft 

platform for the military installation served

Addressed

(ix) a determination of whether the civilian hospital system in which the 

facility resides is a federally designated underserved medical community 

and the effect on such community from any reduction in staff or functions 

or downgrade of the facility

Addressed

(x) If the facility serves a training center—

(I) a determination of the risk with respect to high-tempo, live-fire 

military operations, treating battlefield-like injuries, and the potential 

for a mass-casualty event if the facility is downgraded to a clinic or 

reduced in personnel or capabilities; and

(II) a description of the extent to which the Secretary, in making such 

determination, consulted with the appropriate training directorate, 

training and doctrine command, and forces command of each military 

department;

(I) Partially addressed: Did not state whether facilities 

were training centers or explicitly speak to battlefield 

type injuries. However, did assess risk to operational 

mission for three unspecified “small” military treatment 

facilities.

(II) Partially addressed: Consultations with numerous 

service stakeholders are noted, but not specifically 

with the organizations listed.

(xi) a site assessment by TRICARE to assess the network capabilities of 

TRICARE providers in the local area; 

Addressed

Appendix II: Comparison of DOD’s Military 

Health System Modernization Study with 

Requirements Listed in 2015 National 

Defense Authorization Act
Appendix II: Comparison of DOD’s Military 

Health System Modernization Study with 

Requirements Listed in 2015 National Defense 

Authorization Act

Page 32 GAO-16-820 Military Health Modernization

National Defense Authorization Act (NDAA) elements and items Our assessment

(xii) the inpatient mental health care availability; and Partially addressed: Only limited data were available.

(xiii) the average annual inpatient care directed to civilian medical 

facilities.

Addressed

(B) For each military medical treatment facility considered under the 

modernization study—

Partially Addressed

(i) the civilian capacity by medical specialty in each catchment area; Addressed

(ii) the distance in miles to the nearest civilian emergency care 

department;

Addressed

(iii) the distance in miles to the closest civilian inpatient hospital, listed by 

level of care and whether the facility is designated a sole community 

hospital;

Addressed

(iv) the availability of ambulance service on the military installation and 

the distance in miles to the nearest civilian ambulance service, including 

the average response time to the military installation; 

Addressed

(v) an estimate of the cost to restructure or realign the military medical 

treatment facility, including with respect to bed closures and the civilian 

personnel reduction; and 

Partially addressed: This figure was not explicitly 

stated, but it was included in the savings estimates.

(vi) if the military medical treatment facility is restructured or realigned, an 

estimate of—

(I) the number of civilian personnel reduction, listed by series;

(II) the number of local support contracts terminated; and

(III) the increased cost of purchased care.

(I) Partially addressed: A net figure was included, but 

not for each medical treatment facility or listed by 

series.

(II) Not addressed: Study stated estimates would be 

made during implementation.

(III) Partially addressed: A net figure was included, but 

not for each medical treatment facility.

(C) The results of the modernization study with respect to the 

recommendations of the Secretary to restructure or realign military medical 

treatment facilities 

Addressed

The Study presented its findings and 

recommendations for modernization of the military 

healthcare system.

(D) An assessment of the analysis made by the Secretary to inform decisions 

regarding the modernization of the military health care system in the 

modernization study

Addressed

The Study documented the analyses used to inform 

the recommendations for modernization.

(E) An assessment of the extent to which the Secretary evaluated in the 

modernization study the impact on the access of eligible beneficiaries to 

quality health 

Partially Addressed

(i) changes in military medical treatment facility infrastructure; Partially addressed: The Study states that the effect 

of potential changes would be considered during 

implementation of recommendations; however, quality 

and satisfaction not addressed in the report.

(ii) changes in staffing levels of professionals; and Partially addressed: The Study states that the effect 

of potential changes would be considered during 

implementation of recommendations; however, quality 

and satisfaction not addressed in the report.
Appendix II: Comparison of DOD’s Military 

Health System Modernization Study with 

Requirements Listed in 2015 National Defense 

Authorization Act

Page 33 GAO-16-820 Military Health Modernization

National Defense Authorization Act (NDAA) elements and items Our assessment

(iii) changes in inpatient, ambulatory surgery, and specialty care capacity 

and capabilities.

Partially addressed: The Study listed the criteria by 

which DOD evaluated the impact of recommended 

changes on beneficiaries’ access to high-quality 

health care and satisfaction with such care; however, 

while institutional quality data could relate to the 

request for impact assessment, it remains unclear 

whether DOD explicitly used the data in this manner. 

(F) An assessment of the extent to which the Secretary evaluated in the 

modernization study how any reduced inpatient, ambulatory surgery, or 

specialty care capacity and capabilities at military medical treatment facilities 

covered by the study would impact timely access to care for eligible 

beneficiaries at local civilian community hospitals within reasonable driving 

distances of the catchment areas of such facilities

Addressed

The Study included data that demonstrated an 

analysis of the effects of potential changes in the 

hospitals on timely access to care for beneficiaries, 

and potential for treatment at nearby civilian hospitals.

(G) An assessment of the extent to which the Secretary consulted in 

conducting the modernization study with community hospitals in locations 

covered by the study to determine their capacities for additional inpatient and 

ambulatory surgery patients and their capabilities to meet additional demands 

for specialty care services

Addressed

The Study noted consultations with local TRICARE 

experts to determine the capacity of community 

hospitals to absorb additional patients.

(H) An assessment of the extent to which the Secretary considered in the 

modernization study the impact that the change in the structure or alignment 

of military medical treatment facilities covered by the study would have on 

timely access by local civilian populations to inpatient, ambulatory surgery, or 

specialty care services if additional eligible beneficiaries also sought access to 

such services from the same providers

Not Addressed

The Study included data on civilian hospital 

capabilities; however, it did not explain the analysis of 

these data that led the Study team to its conclusions 

concerning access to care.

(I) An assessment of the impact of the elimination of health care services at 

military medical treatment facilities covered by the modernization study on 

civilians employed at such facilities

Partially Addressed

The Study noted the number of civilian full-time 

equivalents that will be downsized, but did not 

otherwise address the effect of these changes on 

civilian personnel.

Source: GAO analysis of DOD data. | GAO-16-820
Appendix III: Comments from the Department 

of Defense

Page 34 GAO-16-820 Military Health Modernization

Appendix III: Comments from the 

Department of Defense
Appendix III: Comments from the Department 

of Defense

Page 35 GAO-16-820 Military Health Modernization
Appendix III: Comments from the Department 

of Defense

Page 36 GAO-16-820 Military Health Modernization
Appendix IV: GAO Contact and Staff 

Acknowledgments

Page 37 GAO-16-820 Military Health Modernization

Brenda S. Farrell, (202) 512-3604 or farrellb@gao.gov

In addition to the contact named above, Lori Atkinson, Assistant Director; 

Rebekah Boone; Patricia Donahue; Mae Jones; Linda Keefer; Carol 

Petersen; Adam Smith; Sabrina Streagle; and Adriel Taslim made key 

contributions to this report.

Appendix IV: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments
Related GAO Products

Page 38 GAO-16-820 Military Health Modernization

Defense Health Care Reform: Actions Needed to Help Ensure Defense 

Health Agency Maintains Implementation Progress. GAO-15-759. 

Washington, D.C.: September 10, 2015.

Defense Health Care Reform: Actions Needed to Help Realize Potential 

Cost Savings from Medical Education and Training. GAO-14-630. 

Washington, D.C.: July 31, 2014.

Military Health System: Sustained Senior Leadership Needed to Fully 

Develop Plans for Achieving Cost Savings. GAO-14-396T. Washington, 

D.C.: February 26, 2014.

Defense Health Care Reform: Additional Implementation Details Would 

Increase Transparency of DOD’s Plans and Enhance Accountability. 

GAO-14-49. Washington, D.C.: November 6, 2013.

Defense Health Care: Additional Analysis of Costs and Benefits of 

Potential Governance Structures Is Needed. GAO-12-911. Washington, 

D.C.: September 26, 2012.

Defense Health Care: Applying Key Management Practices Should Help 

Achieve Efficiencies within the Military Health System. GAO-12-224. 

Washington, D.C.: April 12, 2012.

Military Personnel: Enhanced Collaboration and Process Improvements 

Needed for Determining Military Treatment Facility Medical Personnel 

Requirements. GAO-10-696. Washington, D.C.: July 29, 2010.

Related GAO Products

(100345)
The Government Accountability Office, the audit, evaluation, and investigative 

arm of Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the 

federal government for the American people. GAO examines the use of public 

funds; evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO 

posts on its website newly released reports, testimony, and correspondence. To 

have GAO e-mail you a list of newly posted products, go to http://www.gao.gov

and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of production and 

distribution and depends on the number of pages in the publication and whether 

the publication is printed in color or black and white. Pricing and ordering 

information is posted on GAO’s website, http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400, 

U.S. Government Accountability Office, 441 G Street NW, Room 7125, 

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814, 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Strategic Planning and 

External Liaison

Please Print on Recycled Paper.