This page left intentionally blank.
Military Health System Review – Final Report August 29, 2014

TABLE OF CONTENTS

EXECUTIVE SUMMARY .......................................................................................................... 1

The Military Health System................................................................................................... 1

Review Methods and Scope................................................................................................... 2

Key Findings.......................................................................................................................... 3

General Findings........................................................................................................... 3

Access to Care............................................................................................................... 4

Quality of Care.............................................................................................................. 5

Patient Safety ................................................................................................................ 6

Recommendations.................................................................................................................. 7

Conclusion ............................................................................................................................. 8

Additional Considerations ..................................................................................................... 9

1. INTRODUCTION ............................................................................................................. 11

Goals, Objectives, Methods, and Limitations of the Review .............................................. 13

Goals........................................................................................................................... 14

Objectives ................................................................................................................... 14

Methods....................................................................................................................... 14

Data Analysis and Limitations.................................................................................... 17

Organization of the Report .................................................................................................. 18

2. OVERVIEW OF DOD’S MILITARY HEALTH SYSTEM ......................................... 19

Introduction.......................................................................................................................... 19

Beneficiary Demographics .................................................................................................. 21

Policies for Priority of Access............................................................................................. 22

DoD Military Medical Operations....................................................................................... 22

Overview of Military Health System (MHS) Governance .................................................. 24

Governance Reform Related to Performance Improvement................................................ 29

Component Responsibilities within the Military Health System................................ 31

Resource Support for Patient Quality, Access and Safety................................................... 32

Conclusions.......................................................................................................................... 32

3. ACCESS TO CARE IN THE MILITARY HEALTH SYSTEM .................................. 33

Introduction.......................................................................................................................... 33

Access to Care Governance ................................................................................................. 34

Policy Review and Identification of Gaps........................................................................... 35

ASD(HA) – Overarching Guidelines.......................................................................... 35

Military – Service-Level Instructions......................................................................... 35

TRICARE – Purchased Care Guidelines.................................................................... 35

External Reviews........................................................................................................ 35

i
August 29, 2014 Table of Contents

Education and Training........................................................................................................ 36

Methodology........................................................................................................................ 36

Direct Care Component Analysis........................................................................................ 38

Acute Care .................................................................................................................. 38

Routine Care ............................................................................................................... 44

Specialty Care ............................................................................................................. 47

Average Days to Third Next Specialty Care Appointment ........................................ 49

Emergency Department and Urgent Care Center Utilization by MTF Enrollees in the 

Purchased Care Network ..................................................................................................... 56

Patient Satisfaction Surveys................................................................................................. 57

Purchased Care .................................................................................................................... 64

Network Adequacy (United States) ............................................................................ 64

28-Day Access to Network Care (United States) ....................................................... 66

Patient Satisfaction with Network Access to Care (United States) ............................ 67

Access to Network Care (Overseas) ........................................................................... 68

Site Visit Information .......................................................................................................... 70

Introduction................................................................................................................. 70

Regional and Headquarters Leadership Survey.......................................................... 71

On-site: Leadership, Staff, Patients ............................................................................ 71

Site Visit Survey Results ............................................................................................ 72

Staff Town Halls......................................................................................................... 73

Beneficiary Town Halls.............................................................................................. 73

Access to Care: Overall Findings and Recommendations................................................... 74

Recommendations to Improve Access to Care .................................................................... 75

4. QUALITY OF CARE IN THE MILITARY HEALTH SYSTEM................................ 77

Introduction.......................................................................................................................... 77

Quality of Care Governance ................................................................................................ 77

Policy Review...................................................................................................................... 78

Army Policy Guidance................................................................................................ 79

Navy Policy Guidance ................................................................................................ 80

Defense Health Agency/National Capital Region Medical Directorate (NCR 

MD) Policy Guidance ........................................................................................... 82

Policy Guidance for Purchased Care .......................................................................... 82

Review of Internal and External Studies on Quality of Care .............................................. 83

Gap Analysis........................................................................................................................ 83

Purchased Care Gap Analysis..................................................................................... 84

Education and Training........................................................................................................ 85

Data Analysis....................................................................................................................... 85

Accreditation and Certification................................................................................... 87

Service-Level Discussion............................................................................................ 87

Facility Type, Location, and Other Discussions......................................................... 88

HEDIS® Measures of Performance............................................................................. 88

ii
Military Health System Review – Final Report August 29, 2014

Quality of Care in the Purchased Care Component.................................................... 92

ORYX® – National Hospital Quality Measures.......................................................... 94

Prevention Quality Indicators (PQI)........................................................................... 98

Thirty-Day Readmissions ......................................................................................... 100

National Perinatal Information Center (2010-2013)................................................. 102

Comparative Measures.............................................................................................. 103

National Surgical Quality Improvement Program (NSQIP®).................................. 110

Inpatient Mortality Measures.................................................................................... 119

Experience of Care Summary ................................................................................... 122

Comparison to External Health Systems .................................................................. 126

Primary Care Manager Continuity............................................................................ 129

Site Visit Information ........................................................................................................ 132

Policy ........................................................................................................................ 133

Leadership................................................................................................................. 133

Quality Improvement Infrastructure ......................................................................... 133

Performance Improvement Efforts ........................................................................... 134

Patient Focus............................................................................................................. 135

Staff Town Halls....................................................................................................... 136

Beneficiary Town Halls............................................................................................ 137

Quality of Care: Overall Findings and Recommendations................................................ 138

5. PATIENT SAFETY IN THE MILITARY HEALTH SYSTEM ................................ 141

Introduction........................................................................................................................ 141

Patient Safety Governance................................................................................................. 141

Measures: Using Data to Drive Change.................................................................... 142

Performance Improvement Initiatives....................................................................... 143

Policy Review.................................................................................................................... 143

DoDI 6025.13 and DoDM 6025.13 .......................................................................... 143

Review of External Reports Regarding Patient Safety...................................................... 146

Education and Training: Patient Safety Program .............................................................. 147

Measures of Safety............................................................................................................. 148

Measures within Direct Care settings................................................................................ 149

Patient Safety Culture Survey................................................................................... 149

PSI #90 Composite for the Military Health from CY 2010-2013 ............................ 153

Healthcare-Associated Infections, CY 2010 to 2013................................................ 159

Sentinel Event (SE) Reporting.................................................................................. 164

Root Cause Analysis................................................................................................. 168

Performance Improvement Root Cause Analysis..................................................... 175

Patient Safety Reporting System (PSRS) ................................................................. 178

Measures within Purchased Care Settings................................................................ 180

Site Visit Information ........................................................................................................ 186

Executive Leadership Session................................................................................... 186

Functional Staff Focus Group................................................................................... 186

General Staff Interviews........................................................................................... 187

iii
August 29, 2014 Table of Contents

Patient Interviews...................................................................................................... 187

Staff Town Hall Results............................................................................................ 187

Beneficiary Town Hall Results................................................................................. 188

Site Visit versus Central Data Comparative Summary............................................. 188

Patient Safety: Overall Findings and Recommendations .................................................. 190

Overarching Recommendations to Improve Patient Safety............................................... 191

6. CONCLUSIONS.............................................................................................................. 193

Access to Care ................................................................................................................... 193

Quality of Care................................................................................................................... 194

Patient Safety ..................................................................................................................... 194

Summary............................................................................................................................ 195

iv
Military Health System Review – Final Report August 29, 2014

LIST OF TABLES

Table 1.1 MHS Review Site Visit Locations................................................................................ 16

Table 2.1 Average Weekly Statistics in the MHS ........................................................................ 19

Table 2.2 Distribution of Beneficiaries across the Military Departments and National 

Capital Region Medical Directorate ........................................................................... 22

Table 3.1 MHS vs. CA Standards Comparison ............................................................................ 38

Table 3.2 Average Number of Days to Third Next Acute Appointment (Primary Care) by 

Service and Percent of 24-Hour Care Appointments.................................................. 43

Table 3.3 Type of Patient-Initiated Clinical Message .................................................................. 54

Table 3.4 Registered Users Initiating a Secure Message to PCM, FY14 ..................................... 54

Table 3.5 CAHPS Benchmarks..................................................................................................... 68

Table 4.1 Numbers of Policies, Instructions, Resources and Manuals Reviewed........................ 78

Table 4.2 Comparison of HEDIS® 2013 Data for MHS Against External Comparison 

Organizations (Health Systems 1, 2, and 3)................................................................ 91

Table 4.3 National Hospital Quality Measures Monitored for Both Direct and Purchased 

Care............................................................................................................................. 93

Table 4.4 TJC Core Measure Sets Monitored for both Direct and Purchased Care ..................... 95

Table 4.5 Composite Measures for 13 ORYX Measures ............................................................. 96

Table 4.6 PQI Indicator Measures................................................................................................ 98

Table 4.7 PQI Compliance Rate by Service (2010 – 2013).......................................................... 99

Table 4.8 Readmission Rates According to Services, 2010 – 2013 ........................................... 100

Table 4.9 Readmission Rate According to Facility Type, 2010 – 2013..................................... 101

Table 4.10 Readmission Rate Comparison to External Health Systems, 2010 – 2013 .............. 101

Table 4.11 National Perinatal Information Center – Executive Summary Measures................. 102

Table 4.12 Perinatal Performance Measures Summary – 2013 Annual Rates........................... 108

Table 4.13 Mortality Measures................................................................................................... 119

Table 4.14 Direct Care Component IQI Condition-specific Mortality Rates for 2010 – 2013 .. 120

Table 4.15 Facility-specific Risk-adjusted Standardized Mortality Ratios for 2013.................. 121

Table 4.16 Ratings of Overseas and U.S. Facilities by OCONUS and CONUS........................ 125

Table 4.17 Rating of Health Care Plan, 2011 – 2013 ................................................................. 126

Table 4.18 Rating of Personal Doctor, 2011 – 2013................................................................... 126

Table 4.19 Health Care Rating, 2011 – 2013.............................................................................. 127

Table 5.1 HSOPS Dimensions.................................................................................................... 150

Table 5.2 Direct Care Component HSOPS Results: Average Percent Positive Responses 

across Dimensions .................................................................................................... 150

Table 5.3 Average Percent Positive Responses Across Dimensions.......................................... 151

Table 5.4 HSOPS Percent Positive Results for Comparing Direct Care 2011 Results to 

Health System 3 Survey............................................................................................ 152

Table 5.5 Direct Care CAUTI by ICU Type, for Total Period, CY10 – CY13.......................... 161

Table 5.6 Direct Care CLABSI by ICU Type, for Total Period CY10 – CY13......................... 162

Table 5.7 Direct Care VAP by ICU Type, for Total Period CY10 – CY13 ............................... 162

Table 5.8 DoD Direct Care and Civilian Health Care Systems HAI Rates................................ 163

Table 5.9 Top 5 Sentinel Events by Year ................................................................................... 165

Table 5.10 Top Five Sentinel Events by Service with Frequency Count, 2010 – 2013 ............. 166

v
August 29, 2014 List of Tables

Table 5.11 Number of RCAs reported to PSAC, DHA, and Health Affairs by FY of Event 

Date........................................................................................................................... 169

Table 5.12 RCAs by Event Type submitted to PSAC, FY10 – FY13 (rank ordered)................ 170

Table 5.13 Level of Harm Results for RCA Investigations by FY and Service, FY10 –

FY13 ......................................................................................................................... 173

Table 5.14 Direct Care SE RCA, Level of Harm Findings, FY11 – FY13 ................................ 174

Table 5.15 System 3 SE RCA, Level of Harm Findings, FY11 – FY13.................................... 175

Table 5.16 Service Identified Source for RCA Classification of Event Type and Total 

Number of PI RCAs.................................................................................................. 176

Table 5.17 Top PI RCAs for DoD Overall, Air Force, Navy, Army, and NCR MD ................. 177

Table 5.18 PSI Rates for Purchased Care Regions Compared to AHRQ National 

Benchmarks, FY 10 – FY13 ..................................................................................... 182

vi
Military Health System Review – Final Report August 29, 2014

LIST OF FIGURES

Figure 1.1 Veterans Health Administration.................................................................................. 13

Figure 2.1 MHS Global Distribution of MHS Direct Care Platform............................................ 24

Figure 2.2 Organizational Structure of the Military Health System within the Department of 

Defense ....................................................................................................................... 26

Figure 2.3 Governance Structure of the Military Health System ................................................. 28

Figure 3.1 Average Number of Days to Acute Appointment – Overall: MHS Access 

Standard < 1 Day ........................................................................................................ 39

Figure 3.2 Average Number of Days to Acute Appointment – By Service: MHS Access 

Standard ≤ 1 Day ........................................................................................................ 40

Figure 3.3 Average Number of Days to Acute Appointment – By Facility Type: MHS 

Access Standard ≤ 1 Day ............................................................................................ 41

Figure 3.4 Average Number of Days to Third Next Acute Appointment (Primary Care) –

Direct Care Component Overall ................................................................................. 42

Figure 3.5 Average Number of Days to Third Next Acute Appointment (Primary Care) –

By Service................................................................................................................... 43

Figure 3.6 Average Number of Days to Third Next Acute Appointment (Primary Care) –

By Type of MTF ......................................................................................................... 44

Figure 3.7 Average Number of Days to Third Next Routine Appointment (Primary Care) 

Overall......................................................................................................................... 45

Figure 3.8 Average Number of Days to Third Next Routine Appointment (Primary Care) –

By Service................................................................................................................... 46

Figure 3.9 Average Number of Days to Third Next Routine Appointment (Primary Care) –

By Type of MTF ......................................................................................................... 46

Figure 3.10 Average Number of Days to Specialty Appointment – Overall: MHS Access 

Standard ≤ 28 Day ...................................................................................................... 47

Figure 3.11 Average Number of Days to Specialty Appointment – By Service: MHS 

Access Standard ≤ 28 Day .......................................................................................... 48

Figure 3.12 Average Number of Days to Specialty Appointment – By Facility Type: 

MHS Access Standard ≤ 28 Day ................................................................................ 49

Figure 3.13 Average Number of Days to Third Next Specialty Care Appointment – Overall..... 50

Figure 3.14 Average Number of Days to Third Next Specialty Care Appointment – By 

Service......................................................................................................................... 50

Figure 3.15 Average Number of Days to Third Next Specialty Care Appointment – By 

Facility Type ............................................................................................................... 51

Figure 3.16 TRICARE On-Line (TOL) Web-Enabled Appointments: MHS Overall.................. 52

Figure 3.17 Number of MTF Enrollees Registered in Secure Messaging – By Service, ............. 53

August 2010 – May 2014.............................................................................................................. 53

Figure 3.18 Nurse Advice Line (NAL) Calls Triaged 28 March to 25 July 2014........................ 55

Figure 3.19 MTF Enrollee Primary Care Workload, by Venue of Care, FY11 – FY14 .............. 56

Figure 3.20 Satisfaction with “Getting Care When Needed” (Service Surveys) Overall, 

FY12 Q1 – FY14 Q1................................................................................................... 57

Figure 3.21 Satisfaction with “Getting Care When Needed” (Service Surveys) – By 

Service*, FY12 Q1 – FY14 Q1................................................................................... 58

vii
August 29, 2014 List of Figures

Figure 3.22 TROSS – Satisfaction with Access to Care – Direct Care, FY11 Q1 – FY13 Q4 .... 59

Figure 3.23 TROSS – Satisfaction with Access to Care – By Service, FY11 Q1 – FY13 Q4..... 59

Figure 3.24 TROSS – Satisfaction with “Seeing Provider when Needed.” Direct Care, 

FY11 Q1 – FY13 Q4................................................................................................... 60

Figure 3.25 TROSS – Satisfaction with “Seeing Provider when Needed,” by Service, Direct 

Care, FY11 Q1 – FY13 Q4......................................................................................... 60

Figure 3.26 HCSDB – Satisfaction with “Getting Care Quickly”, FY10 Q1 – FY14 Q2............ 61

Figure 3.27 HCSDB – Satisfaction with “Getting Care Quickly” – by Service, FY10 Q1 –

FY14 Q2..................................................................................................................... 62

Figure 3.28 HCSDB – Satisfaction with “Getting Care When Needed”, FY10 Q1 – FY14 

Q2................................................................................................................................ 63

Figure 3.29 HCSDB – Satisfaction with “Getting Care when Needed” – by Service, 

FY10 Q1 – FY14 Q2................................................................................................... 63

Figure 3.30 Primary Care Providers per 1,000 Enrolled Beneficiaries (Restricted to 

beneficiaries living in Prime Service Areas), FY11 Q1 – FY13 Q3 .......................... 65

Figure 3.31 Specialty Care Providers per 1,000 Enrolled Beneficiaries (Restricted to 

beneficiaries living in Prime Service Areas), FY11 Q1 – FY13 Q3 .......................... 65

Figure 3.32 Percent of Patients in a PSA Required to Drive More than 60 Minutes to see a 

Specialist..................................................................................................................... 66

Figure 3.33 Percentage of Network Referrals Meeting 28 Day Access Standard, by PSA 

(Each point represents one PSA), FY11 Q1 – FY14 Q4 ............................................ 67

Figure 3.34 U.S. Network Satisfaction: Four Satisfaction Measures, FY10 Q1 – FY14 Q1 ....... 68

Figure 3.35 Satisfaction with International SOS Coordination of Specialty Care, FY11 Q2 

– FY13 Q4................................................................................................................... 69

Figure 3.36 Satisfaction with Civilian Network Care, FY11 Q2 – FY13 Q4............................... 70

Figure 3.37 Access: Perceptions Among Regional Headquarters, MTF Leaders, Subject 

Matter Experts (SMEs), Staff Members and Patients During Seven MHS Site 

Visits, 2014 ................................................................................................................. 73

Figure 4.1 Purchased Care Compliance Rate, FY09 – FY12 ....................................................... 94

Figure 4.2 Annual Rate of Vaginal Deliveries Coded with Shoulder Dystocia, CY10 –

CY13......................................................................................................................... 104

Figure 4.3 Annual Postpartum Hemorrhage Rate, CY10 – CY13.............................................. 104

Figure 4.4 Annual Rate of PSI 17 Injury to Neonate, CY10 – CY13......................................... 106

Figure 4.5 2013 Direct Care Breakout of Inborn Birth Trauma Occurrences by Diagnostic 

Code .......................................................................................................................... 106

Figure 4.6 Infant Mortality Rate (per 1,000 live births) for Infants Weighing 500 Grams or 

Greater, CY10 – CY13 ............................................................................................. 107

Figure 4.7 ALL-CASE Mortality and Morbidity, MTFs, 2010 – 2013...................................... 112

Figure 4.8 Morbidity Trends across MTFs, 2010 – 2013........................................................... 113

Figure 4.9 MTF Comparison by Type of Morbidity Outcome, 2010 – 2013............................. 114

Figure 4.10 Service-Level Comparison, 2010 – 2013 ................................................................ 115

Figure 4.11 Number of MTFs that Need Improvement, Meet Standards, or Exceed 

Standards in Post-surgical Morbidity by Facility Type (2010 – 2013) .................... 115

Figure 4.12 MHS Surgical Mortality and Morbidity Rates July 2010 – June 2013 ................... 116

Figure 4.13 MHS and HS3 Surgical Morbidity Rates, July 2012 – June 2013 .......................... 117

viii
Military Health System Review – Final Report August 29, 2014

Figure 4.14 Direct Care Partnership for Patients (PfP) Surgical Site Infection Rates, 2010 – 

2013........................................................................................................................... 118

Figure 4.15 Comparison of TRISS to Three Other Health Systems for Hospital Ratings, 

CY13, Monthly ......................................................................................................... 127

Figure 4.16 Comparison of TRISS to Three Other Health Systems for Recommend 

Hospital, CY13, Monthly .......................................................................................... 128

Figure 4.17 MHS Primary Care Manager (PCM) Continuity..................................................... 130

Figure 4.18 Percent of Appointments where the Patient Saw their Assigned PCM – By 

Service....................................................................................................................... 130

Figure 4.19 Percent of Appointments where the Patient Saw their Assigned PCM – by 

Facility Type ............................................................................................................. 131

Figure 4.20 Perceptions Among Regional Headquarters, MTF Leaders, Subject Matter 

Experts (SMEs), Staff Members and Patients During Seven MHS Site Visits, 

2014........................................................................................................................... 132

Figure 5.1 MTF Performance versus Reference Population, CY10 – CY13 .............................. 155

Figure 5.2 MTF Performance versus National Benchmark Rate, CY10 – CY13 ....................... 156

Figure 5.3 Boxplot of PSI #90 Composite: Direct Care Relative to Systems 1, 2, 3 ................. 158

Figure 5.4 Interval Plot of PSI #90 Composite by System and Time Period ............................. 158

Figure 5.5 Number of SEs across Direct Care, Health System 2, and System 3, FY11 – 

FY13 ......................................................................................................................... 167

Figure 5.6 SE Rates per 1,000 Discharges, Direct Care and Health System 2, FY11 – FY13 ... 167

Figure 5.7 Air Force Top 4 Event Types for RCA Reports Submitted, FY10 – FY13 .............. 171

Figure 5.8 Army Top 4 Event Types for RCA Reports Submitted, FY10 – FY13 .................... 172

Figure 5.9 Navy Top 4 Event Types for RCA Reports Submitted, FY10 – FY13 ..................... 172

Figure 5.10 PI Service RCAs by Year and Service .................................................................... 176

Figure 5.11 Total PSR Events by Month, FY12 – FY13 ............................................................ 178

Figure 5.12 Events by Harm by Month, FY12 – FY13 .............................................................. 179

Figure 5.13 Total Number of Quality Issues (QIs) for AHRQ PSIs, HACs, SREs Identified 

in FY10 – FY13 for Purchased Care ......................................................................... 183

Figure 5.14 Total Number of Quality Issues (QIs) for AHRQ PSIs, HACs, SREs Identified 

in FY10 – FY13, by Region for Purchased Care ...................................................... 184

Figure 5.15 Total Number of National Quality Forum Serious Reportable Events in FY10 – 

FY13, by Region for Purchased Care ....................................................................... 185

Figure 5.16 Safety: Perceptions Among Regional Headquarters, MTF Leaders, Subject 

Matter Experts (SMEs), Staff Members, and Patients During Seven MHS Site 

Visits, 2014 ............................................................................................................... 189

ix
August 29, 2014 List of Figures

This page left intentionally blank. 

x 
Military Health System Review – Final Report August 29, 2014

EXECUTIVE SUMMARY 

On May 28, 2014, the Secretary of Defense ordered a comprehensive review of the Military 

Health System (MHS). The review was directed to assess whether: 1) access to medical care in 

the MHS meets defined access standards; 2) the quality of health care in the MHS meets or 

exceeds defined benchmarks; and 3) the MHS has created a culture of safety with effective 

processes for ensuring safe and reliable care of beneficiaries. This is the first time the MHS has 

taken an enterprise view of such scope in these areas.

Based on information analyzed during the review, 

the MHS provides good quality care that is safe and 

timely, and is comparable to that found in the 

civilian sector. However, the MHS demonstrates 

wide performance variability with some areas 

better than civilian counterparts and other areas 

below national benchmarks. 

Together, the review’s results and the professional 

inputs from six external experts indicate clear 

opportunities to improve health care delivery. By 

implementing effective strategies used by other 

high-performing organizations, the MHS can create 

an optimal health care environment that focuses on 

continuous quality improvement where every 

patient receives safe, high-quality care at all times.

The major recommendations in this report are 

directed at system enhancements to address areas of 

concern and to drive change that will foster creation of a high reliability health system. High 

reliability organizations, in general, are those where harm prevention and quality improvement 

are second nature to all in the organization. Such organizations recognize the risk of over 

simplification in complex systems: thus, implementation of the proposed recommendations 

should not be expected to result in immediate change. MHS governance can support 

performance improvement with better analytics, greater clarity in policy, and aligned training 

and education programs. However, improving outcomes is about decreasing performance 

variance at the individual facility level, which requires consistent leadership vigilance, with the 

goal of making the MHS a top-tier health care system.

The Military Health System

The MHS is a global, comprehensive, integrated system that includes combat medical services, 

health readiness futures, a health care delivery system, public health activities, medical education 

and training, and medical research and development. The fundamental mission of the MHS, 

providing medical support to military operations, is different from that of any other health 

system in the United States. The operational aspects of the MHS are divided among the three 

“The report provided no evidence 

of substantive deficiencies in the 

safety, quality, and access to care 

at MHS that would warrant broad 

and urgent changes.”

Peter Pronovost, M.D., Ph.D., 

FCCM

Johns Hopkins Medicine Senior 

Vice President for Patient Safety 

and Quality 

Director of the Armstrong Institute 

for Patient Safety and Quality

1 
August 29, 2014 Executive Summary

Military Departments (Army; Navy, to include Marine Corps; and Air Force), with each Service 

and the Defense Health Agency controlling and operating their own medical centers, hospitals, 

and clinics worldwide. 

As one of the largest health care providers in the United States, the MHS combines resources 

from both direct and purchased care components, facilitating ready access to health care for 9.6 

million beneficiaries, including Service members, retirees, and their eligible family members. In 

Fiscal Year 2013, the direct care component of military treatment facilities (MTFs) consisted of 

56 hospitals, 361 ambulatory care clinics, and 249 dental clinics, operating worldwide and 

employing 60,389 civilians and 86,051 military personnel. The purchased care component, 

which is used when care cannot be provided within the military system, includes civilian 

network hospitals and providers operated through TRICARE regional contracts. 

Like every large health care system, the MHS is constantly responding and adapting to changing 

demographics, shifting policies, evolving standards for access and quality, advances in science 

and medicine, complex payment and cost considerations, rapidly evolving communications and 

information technology capabilities, and fluid patient expectations. In addition, the MHS 

recently reformed its governance structure in October 2013. All health care systems, including 

the MHS, are expected to engage in systematic performance reviews designed to assess new 

developments and to measurably improve the delivery of health care services and the health 

status of the population served. These factors combined warranted an assessment of the general 

state of care in the MHS in order to determine where improvement is possible. 

Review Methods and Scope

The intent of the MHS review was to establish a baseline measure of MHS performance and to 

determine if that performance is comparable to top performing health care systems. The Deputy 

Secretary of Defense chartered a Department of 

Defense (DoD) working group, with substantial input 

from individual experts outside of DoD, to conduct 

this review (members are listed in Appendix 8). This 

review was also tasked with identifying gaps that 

prevent the MHS from being considered a leading 

health care system, and with offering 

recommendations to facilitate progress. 

Over a two-month period, subject matter experts 

collected and analyzed a variety of current metrics, 

compared them to existing national standards, and 

validated them by visiting selected military hospitals 

and clinics. The working group reviewed previous 

reports on the performance of the MHS with regard to 

access, quality, and safety and documented 

compliance with those recommendations. In 

addition, it reviewed all relevant policies—both 

“The thoroughness of the 

approach to the Report was 

apparent in its inclusion of 

multiple data sources, 

stakeholders, and analysis 

methods.” 

Katherine L. Kahn, M.D. 

Senior Scientist, RAND

Professor of Medicine 

David Geffen School of 

Medicine 

University of California, Los 

Angeles 

2 
Military Health System Review – Final Report August 29, 2014

Service specific and issued by the Office of the 

Secretary of Defense. Three external health care 

systems provided their data to the MHS for the 

expressed purpose of comparison. During town hall 

gatherings of patients and staff at seven MTFs, the 

working group obtained impressions of how well the 

system provides timely access to health care, and the 

quality and safety of the care delivered. The collected 

information, methodology, and subsequent 

recommendations were reviewed by external experts 

to ensure that the review was comprehensive, the data 

honestly represented, and the conclusions, based on 

data analyses, were valid1. 

All external reviewers acknowledged the challenge of comparing performance across health 

systems and noted that many of the challenges facing the MHS are similar to inherent challenges 

throughout U.S. health care.

Due to the restricted time for the review, not all areas of interest were investigated; many of 

these are noted in the report and by the external experts. For example, determining access for 

individuals with specific clinical conditions would provide additional information, but could not 

be completed in the time available. Other areas of special interest identified in the review are 

documented in the report and will need further evaluation. 

Key Findings 

The full analysis and findings of the review are found in the report and appendices. The findings 

fall into two categories, as summarized below: general findings that apply across the areas of 

access, quality, and safety, and findings specific to each area of concern.

General Findings

The new MHS governance structure has resulted in significant gains in terms of collaboration 

and alignment among the Services and the Defense Health Agency (pages 24-31). However, no 

single set of metrics is used across the enterprise to monitor performance in the areas of access, 

quality, and safety, nor are there performance reviews of the system as a whole in these areas. 

Moreover, the purchased care component is not aligned with the direct care component in terms 

of data collected or metrics used, making it difficult to draw comparisons between the two 

components. 

1 For this review, external reviewers participated as individual experts in their personal capacities, and not as the 

employees or representatives of their affiliated institutions.

“... , it is not possible to produce 

clinical quality or patient safety 

measures that can accurately rank 

care.”

Brent C. James, M.D., M.Stat.

Chief Quality Officer and

Executive Director Intermountain 

Institute for Health Care Delivery 

Research

3 

 
August 29, 2014 Executive Summary

The review identified a major gap in the ability of the MHS to analyze systemwide health care 

information. Although the MHS has a wealth of data, the ability to analyze those data and use 

the results to guide decision making in quality and patient safety is nascent. Differences in 

interpretation of policy result in data incompatibility, which adds to the challenge. Without a 

common set of metrics, it is difficult to present systemwide data in a coherent fashion. 

Transparency goes hand in hand with a culture of safety, with a lack of transparency being the 

result of multiple factors. Finally, lack of a mechanism to recognize patient input at the 

enterprise level makes it difficult to act on feedback as to what the patient would find beneficial. 

Although leadership and the local subject matter experts in the MTFs have a working knowledge 

of desired behaviors to promote a culture of safety, the same cannot be said uniformly about 

frontline clinical personnel. 

Access to Care 

On average, access to care meets the identified standards; however, performance varies across 

the system and purchased care data are incomplete. For example, in the direct care component, 

the average number of days for TRICARE 

Prime patients to obtain an appointment to a 

specialty care provider is 12.4 days (range 6 to 

22 days), well under the identified standard of 

28 days (pages 47-49). Access to an 

appointment for patients who need immediate, 

but not emergency care, averages less than the 

24-hours standard for most of the direct care 

health facilities, but 11 do not meet the 

established access standard. Comparable 

purchased care data are not available, primarily 

due to alternative access measures defined by 

contract specifications, leaving a sizable blind 

spot for understanding access in the purchased 

care component. 

One important finding was the notable 

difference between data that reflect compliance with access standards and the reported 

satisfaction of patients with their ability to receive timely care in MTFs (pages 57-63). This 

issue will require additional study in order to understand the cause of this discrepancy. 

A review of current policies showed that there is no MHS measure for evaluating office waiting 

times, an existing standard (pages 35-36). This deficiency was also noted for purchased care. 

In addition to face-to-face encounters, the direct care component has other methods for accessing 

care, including secure messaging, web-enabled appointment booking, and the Nurse Advice Line 

(pages 52-55). These newer approaches will require ongoing monitoring to ensure that they are 

functioning as designed. 

“Research indicates that using high tech 

technology and ‘secure messaging’ can 

improve access and quality of care, 

reduce medical cost, and improve 

patient satisfaction.” 

Qi Zhou, M.D. 

Executive Director

Performance Measurement Program 

Strategy

& Quality Programs Oversight

Blue Cross Blue Shield of 

Massachusetts

4 
Military Health System Review – Final Report August 29, 2014

Quality of Care

Overall, the review of quality measures showed mixed results. Although there are areas in which 

the MHS excels, there is considerable variation across the system, both for specific clinical 

measures and for individual MTFs. Additionally, there is a general deficiency of data 

concerning clinical quality and outcome measures for care provided in the purchased care 

component. 

All direct care component hospitals and clinics are accredited or certified by external agencies 

(pages 87-88). This provides a certain level of quality and safety assurance for patients and 

allows systems to objectively identify areas for performance improvement. In addition to 

seeking and obtaining accreditation and certification as an indicator of quality, the MHS has 

identified several nationally recognized health care quality performance measures and, unlike the 

private sector, mandates reporting on these measures by every direct care health facility, where 

appropriate. 

HEDIS® measures (which assess outpatient 

preventive services and health outcomes) showed 

high variability across the MHS (pages 88-92). The 

HEDIS® measures chosen by the MHS for 

monitoring quality are selected to drive improved 

outcomes in specific areas. Once the MHS meets 

and sustains the desired target, the measure is 

“retired,” the result being that current measures will 

skew toward underperformance. 

Of the 18 HEDIS® measures monitored by the 

MHS, three were below the 25th percentile, and 

seven were between the 25th and 50th percentile. In 

2013, 10 of the 18 measures showed statistically 

significant improvement, while 6 of the 18 measures 

showed statistically significant decline. Only 12 HEDIS® measures are monitored for the 

purchased care component; 11 of these are less than the NCQA 75th percentile benchmark. 

Hospital quality performance as measured by The Joint Commission’s ORYX data demonstrates 

a similar spectrum of results (pages 94-98). The MHS direct care component meets or exceeds 

target levels for a majority of measures, but needs improvement in a significant number of areas. 

In comparison, the purchased care component collects data for only 5 out of the 13 measures 

reported by the direct care component. This highlights the difficulty of making reliable 

comparisons of performance between direct care and purchased care, and among hospital 

systems in general.

National Perinatal Information Center (NPIC) data show that the direct care component has 

statistically lower rates of infant mortality and maternal trauma than the NPIC averages (NPIC’s 

benchmark is comprised of 86 high-volume obstetric care hospitals) (pages 102-110). However, 

“Overall, MHS performance 

mirrors what we see in the private 

sector, a good deal of mediocrity, 

pockets of excellence, and some 

serious gaps.” 

Janet M. Corrigan, Ph.D. 

Distinguished Fellow 

The Dartmouth Institute for 

Health Policy and Clinical 

Practice

5 
August 29, 2014 Executive Summary

on other measures (to include postpartum hemorrhage and undefined neonatal trauma), the MHS 

is performing statistically worse than the NPIC averages. In addition to the potential quality of 

care issues deserving further examination, administrative coding issues may confound the 

understanding of observed outcomes. Further review of individual clinical areas and specific 

facilities is required to determine the cause or causes of variance. 

The National Surgical Quality Improvement Program (NSQIP), sponsored by the American 

College of Surgeons, collects voluntarily submitted 

risk-adjusted data from approximately 400 hospitals 

and compares the data against performance metrics 

for surgical morbidity and mortality. Of the 56 

inpatient DoD MTFs, 17 facilities who met the 

volume criteria voluntarily participate in NSQIP. 

The MHS does not currently require participation in 

this program.

Surgical mortality (death rate) is within the expected 

range at all 17 DoD MTFs that participate in NSQIP 

(pages 110-119). Surgical morbidity (surgical 

complication rate) was statistically higher than 

expected in 8 of 17 participating MTFs in 2013 and 

there was persistent poor performance in three 

MTFs. Three of 17 MTFs in the most recent 

reporting period are performing at the top tier 

nationally. Of note, only 10 percent of U.S. 

hospitals participate in the NSQIP and this may 

represent a unique subset of health care systems that 

are leading the way in high-quality surgical care. 

Patient Safety

The MHS culture of safety is comparable to that found in the civilian sector based on averages 

from nationally standardized surveys of employee perceptions and patient response rates (pages 

149-153). The MHS had lower averages in 5 of the 12 domains in the national Hospital Survey 

on Patient Safety Culture; staffing, teamwork within units, and organizational learning were of 

greatest concern. 

The execution and content of root cause analysis (RCA) to understand the possible causes of 

adverse health events related to care (sentinel events) remains highly variable across the Services 

(168-175). In addition, there has been a failure to routinely follow up on reported RCAs to 

ensure that systemic issues identified were corrected. 

The MHS has improved on measures for many hospital-acquired conditions through the national 

Partnership for Patients program (pages 160-164). Select safety measures, however, remain

higher than average among MTFs compared to other health care systems (for example, central 

“Until rank and file internalize

their roles in promoting safety 

and preventing harm, 

performance will be mediocre. 

Leadership must declare and 

then demonstrate their 

commitment to a culture that 

encourages reporting, is not 

punitive, and is dedicated to 

improvement.” 

Pamela F. Cipriano, Ph.D., 

R.N., NEA-BC, FAAN

President

American Nurses Association

6 
Military Health System Review – Final Report August 29, 2014

line-associated bloodstream infection rates should have low rates with a goal of zero incidents). 

There is also no comprehensive plan to standardize requirements for monitoring device-related 

infections, such as those related to a catheter. 

Fewer than 30 percent of staff actively reports patient safety events as identified by results from 

the 2011 Hospital Survey on Patient Safety Culture (pages 178-180). The Patient Safety 

Reporting System used to report patient safety events is not designed to record harm rates. 

Overall, the reviewers could not validate that current processes provide an accurate indication of 

the MHS' level rate of harm. 

Recommendations

The following six major recommendations are based on review findings, supported by data, and 

validated by external review. In the body of the main report, additional recommendations within 

the Access, Quality, and Safety sections define specific action steps for performance 

improvement. 

I. Take immediate action to improve underperformance

Recommendation: The MHS should identify the cause of variance for MTFs that are outliers for 

one or more measures and, when due to poor performance, develop corrective action plans to 

bring those MTFs within compliance. 

II. Establish clear enterprise performance goals with standardized metrics and hold the 

system accountable for improvement

Recommendation: The MHS should develop a 

performance management system adopting a core 

set of metrics regarding access, quality, and patient 

safety; further develop MHS dashboards with 

systemwide performance measures; and conduct 

regular, formal performance reviews of the entire 

MHS, with the Defense Health Agency monitoring 

performance and supporting MHS governance 

bodies in those reviews.

III. Make good decisions by relying on 

accurate data

Recommendation: The MHS should develop an 

enterprise-wide quality and patient safety data analytics infrastructure, to include health 

information technology systems, data management tools, and appropriately trained personnel. 

There should be clear collaboration between the Defense Health Agency’s analytic capabilities, 

which monitor the MHS overall, and the Service-level analytic assets.

“The foundation for achieving top 

performance is already in place and 

is being enhanced with new 

approaches that will provide for 

system wide goals, measures, and 

review of performance.” 

Pamela F. Cipriano, Ph.D., R.N., 

NEA-BC, FAAN

President

American Nurses Association

7 
August 29, 2014 Executive Summary

IV. Show information to everyone – patients, providers, and policy makers 

Recommendation: The MHS should emphasize transparency of information, including both the 

direct and purchased care components, with visibility internally, externally, and to DoD 

beneficiaries. Greater alignment of measures 

for the purchased care component with those of 

the direct care component should be 

incorporated in TRICARE regional contracts.

V. Drive the necessary change with MHS 

governance 

Recommendation: Through MHS governance, 

policy guidance can be developed to provide the 

Services with common executable goals. While 

respecting the Services’ individual cultures, this 

effort would advance an understanding of the 

culture of safety and patient-centered care across 

the MHS. 

VI. Leverage common standards and processes to facilitate improvement

Recommendation: The MHS should continue to develop common standards and processes 

designed to improve outcomes across the enterprise in the areas of access, quality, and patient 

safety where this will improve quality, or deliver the same level of quality at decreased cost (i.e., 

better value). 

Conclusion

The findings and recommendations in this report provide an approach for improving the 

performance of the MHS. Appendix 6.1 includes an action list and timelines for execution. 

Recommended actions are divided into those that can be acted on immediately, those that require 

the development of more integrated action plans, and those that require further study to permit 

comprehensive analysis and consideration of the information. 

In addition, within three months of the completion of this report, the MHS will review the 

possible reasons why specific facilities are significantly underperforming on one or more 

measures. When variance is due to poor performance, a corrective action plan will be developed 

and submitted, taking into consideration the unique aspects of those facilities. 

The foundation for improving performance in the MHS rests on combining the concepts of an 

integrated health care system with those of high reliability organizations. The MHS must 

continue to mature as an integrated health system, improving alignment among the Services and 

between the direct care and purchased care components, and placing particular emphasis on 

improving transparency related to access, quality of care, and patient safety. The principles of a 

“Overall, the results are mixed. MHS 

meets or exceeds many internal and 

external standards and benchmarks in 

the areas of access, quality, and safety, 

but there is variability within MHS 

and some performance gaps.” 

Janet M. Corrigan, Ph.D. 

Distinguished Fellow 

The Dartmouth Institute for Health 

Policy and Clinical Practice

8 
Military Health System Review – Final Report August 29, 2014

high reliability organization are operationalized through leadership engagement, a culture of 

quality and safety, robust process improvement through regular performance reviews, adoption 

of industry best practices, and minimization of undesirable variation across the system. These 

efforts should be linked to Service strategies, which may require revision of current policies. 

The high-level recommendations offered in this report, if implemented, will constitute major 

steps along the path to a high reliability organization.

Additional Considerations

For readers without a background in health care and statistics, there are caveats that should be 

considered when interpreting the data presented in this report. First, the review is an “as is” 

assessment based on available data whenever possible. Furthermore, in some cases the data were 

collected or aggregated differently than had previously been done at the facility or Service level. 

As is the case when looking at systems as large as the MHS, there are potential issues with 

conflicting data points, data integrity, and incomplete data. 

An example of conflicting data points is in the area of access, where current access measures 

suggest that the direct care component compares very favorably to civilian care and yet the 

patient satisfaction data indicate that patients are more satisfied with access in the purchased care 

component. It takes time and effort to ensure data validity and accuracy in a system as large as 

the MHS, and further assessment is required. The same can be said of those areas where the data 

are incomplete. This was a particular challenge in attempting to assess the purchased care 

component. The ability of the MHS to evaluate the quality of care is dependent on the data 

provided by civilian providers. This is a major finding of the report and is addressed in the 

recommendations. 

Finally, caution is advisable when using the data to assess where the MHS stands compared to 

U.S. health care in general, or against specific systems. There is no standardized data set used to 

evaluate health systems. The report demonstrates this fact in its attempt to compare the MHS 

with three premier U.S. health systems. Of the access, quality, and patient safety measures used 

in this review, no single measure was directly comparable across all four systems. As a result, 

the review used national benchmarks, where available, and other standards when a national 

benchmark could not be found. It is illustrative to note that most reporting of data regarding 

health care quality and patient safety is voluntary in the civilian sector. By participating in these 

initiatives, those hospitals and health systems have demonstrated a commitment to excellence 

that is above the norm. 

For all of the above reasons, this report should be considered a step in the journey for the MHS, 

rather than an endpoint. Although the recommendations provide a clear path forward, further 

questions raised in this effort will be answered by more in-depth analysis in multiple areas. As 

has been emphasized throughout this summary, health systems are complex, and it would be 

unreasonable to expect that all of the answers to the questions raised as a result of this review 

would be found in 90 days. 

9 
August 29, 2014 Executive Summary

This page left intentionally blank.

10
Military Health System Review – Final Report August 29, 2014

1. INTRODUCTION

The Military Health System (MHS) is a global, comprehensive, integrated system that includes 

combat medical services, peacetime health care delivery, public health services, medical 

education and training, and medical research and development. The Department of Defense 

(DoD) aims to ensure that all active and retired members of the military, as well as their families, 

receive prompt, safe, high-quality care at all times. As one of the largest health care systems in 

the United States, with total spending of more than $50 billion per year, the MHS includes both a 

direct care component, composed of DoD-operated and staffed health care facilities, and a 

purchased care component operated through TRICARE regional contracts.

The MHS shares many features with civilian health care systems. Like every large health care 

system, it is constantly responding and adapting to changing demographics, shifting policies, 

evolving standards for access and quality, advances in science and medicine, complex payment 

and cost considerations, rapidly evolving communications and information technology 

capabilities, and fluid patient expectations. The MHS serves diverse populations in every 

imaginable health care setting. It is unique in that it is structured and operationalized through an 

extensive array of statutory requirements, instructions, policies, and guidelines of DoD, the 

Military Departments (or Services, to include Army, Navy [including Marine Corps], Air Force), 

as well as TRICARE, the Department’s health care benefits program. Moreover, it does not 

operate on a traditional reimbursement system as found in the civilian sector, and is subject to 

congressional authorization and appropriation processes that direct its activities and use of 

resources.

The MHS has faced multiple distinctive challenges over the past decade, including supporting 

deployment of a medically ready force fighting two wars, reorganization of governance structure, 

implementation of enterprise-wide common business processes, and creation of shared services 

in an integrated delivery system. Against the backdrop of an ever-changing health care 

landscape are new regulatory stipulations, increased security requirements, budgetary pressures, 

and base realignment and closure procedures. With each challenge, MHS leadership has 

responded, taking action to address opportunities and mitigate risks.

All health care systems, including the MHS, are expected to engage in systematic processes that 

lead to measurable improvement in health care services and the health status of the population 

served. From a systems perspective, the emphasis in the civilian sector is increasingly on the use 

of data to drive decision-making by identifying areas of variance and opportunity. In the areas of 

patient safety, quality, and access, this paradigm revolves around leadership’s use of data 

analysis to drive process improvement. While significant gains have been made across U.S. 

health care, coordinated efforts are hindered by a lack of comparative data and accepted 

benchmarks, particularly in the areas of patient safety and access. The challenges inherent in 

these efforts are further discussed in the sections of this report comparing the MHS with similar 

civilian systems, but are mentioned here to highlight that the MHS is on the same journey as 

other top-level health systems in the United States; that is, driving toward the goal of becoming a 

high-performing, high reliability organization. 

11
August 29, 2014 1. Introduction

The Joint Commission2 officers Chassin and Loeb describe specific initiatives that health care 

organizations can take to reduce errors and improve patient safety in a highly reliable fashion. 

Key components of these efforts include commitment of organizational leadership to patient 

safety and zero tolerance of harm, development of a functional culture of safety throughout the 

organization, and widespread deployment of process improvement tools.3 The MHS continually 

monitors processes and outcome measures to assess the quality of clinical care provided to 

enrolled beneficiaries (see Section 4 on Quality of Care for details). Just as the MHS often leads 

the nation in health education, training, research, and technology, it also must lead in efforts to 

consistently deliver reliable performance and constantly improve quality and safety with each 

care experience.

On May 28, 2014, the Secretary of Defense ordered a comprehensive review of the MHS, with a 

specific focus on health care access, patient safety, and quality of care in both the direct care 

component and the purchased care component (see Appendix 1.1). The review was conducted 

by subject matter experts in the Services and the Defense Health Agency (DHA), with input from 

outside experts in the areas of patient safety and health care quality. 

The scope of this review does not include health care provided in support of the Combatant 

Commands and deployed operational forces. Care provided to the Nation’s Armed Forces in the 

course of military operations is widely considered to be world class and cutting edge and has 

been the subject of other reviews.4

 Moreover, the policies and organizational structures 

governing health care provided during military operations differ significantly enough from the 

nonoperational setting to warrant exclusion from this review. In addition, this review does not 

include dental care, wounded warrior care (which is subject to several ongoing mandated 

reviews), or beneficiaries enrolled in TRICARE Standard or Extra.5

. (See Appendix 1.2 for the 

Terms of Reference for this review.) Finally, this review does not include the health care system 

serving our Nation’s Veterans, which is administered separately through the Department of 

Veterans Affairs (see Figure 1.1). 

2 The Joint Commission accredits and certifies more than 20,500 health care organizations and programs in the 

United States. Joint Commission accreditation and certification is recognized nationwide as a symbol of quality that 

reflects an organization’s commitment to meeting certain performance standards.

3 Chassin MR, Loeb JN. High reliability health care: Getting there from here. The Milbank Quarterly. 2013; 

91(3):459-490. 4 See, for example: U.S. Central Command, Pre-Hospital Trauma Care Assessment Team. Saving Lives on the 

Battlefield: A Joint Trauma System Review of Pre-Hospital Trauma Care in Combined Joint Operating AreaAfghanistan (CJOA-A) Final Report. January 30, 2013; Office of the Surgeon General, Department of the Army, 

United States of America. Combat Casualty Care: Lessons Learned from OEF and OIF. Fort Detrick, Maryland, 

2012; Department of Defense. Guidance for the Development of the Force FY 2010-2015. 5 TRICARE includes a variety of insurance-like arrangements for health care services. Beneficiaries can select 

which product best meets their needs.

12

 
Military Health System Review – Final Report August 29, 2014

Figure 1.1 Veterans Health Administration

Veterans Health Administration

Some military Veterans are eligible to receive care through a separate health care system than the 

MHS—the Veterans Health Administration (VHA), part of the Department of Veterans Affairs

(VA). A Veteran is someone who has served in the active military, naval, or air service, and was 

discharged or released from service under conditions other than dishonorable, as specified in 38 

U.S.C. 101(2). Active service includes full-time duty in the National Guard or a Reserve 

component, other than full-time duty for training purposes. 

Veterans have a different status than those who retire from military service. A military retiree is 

any former member of the uniformed services who is entitled, under statute, to retired,

retirement, or retainer pay. Examples include, but are not necessarily limited to, spending 20 or 

more years in the military or permanent retirement by reasons of physical disability. All military

retirees are Veterans, but not all Veterans are military retirees. Some military retirees can 

receive care at VA facilities in addition to MHS facilities.

The number of Veterans who can be enrolled in the VHA health care program is determined by 

the amount of money Congress provides to VA each year. Since funds are limited, VA 

established eight priority groups to ensure that certain groups of Veterans can be enrolled before 

others. For example, highest priority is given to Veterans with Service-connected disabilities 

rated 50 percent or more and Veterans assigned a total disability rating for compensation based 

on unemployability.

VHA is an integrated health care system divided into 21 Veterans Integrated Service Networks, 

or VISNs — regional systems of care working together to better meet local health care needs and 

provide greater access to care. VHA consists of 150 medical centers, nearly 1,400 communitybased outpatient clinics, community living centers, Vet Centers, and Domiciliaries. The 

combination of VA health care facilities and the more than 53,000 independent licensed health 

care practitioners who work within them provide comprehensive care to more than 8 million 

enrolled Veterans each year. The VHA has a medical care budget of more than $55 billion 

annually.

6

Goals, Objectives, Methods, and Limitations of the Review

This review was conducted with the goals of assessing three central aspects of a quality health 

care system and developing recommendations for improved performance across the MHS. 

Objectives and methods for meeting these goals are summarized below. Greater detail on 

methods is provided in each of the subsequent sections regarding access, quality, and safety. 

6 U.S. Census Bureau, American Community Survey PUMS, (2011). Prepared by the National Center for Veterans 

Analysis and Statistics.

13

 
August 29, 2014 1. Introduction

Goals

Access to Care: To determine if the MHS provides ready access to medical care as defined 

by access standards in policies and guidance of the Office of the Assistant Secretary of 

Defense for Health Affairs (HA) and the military medical departments, and in TRICARE 

contract specifications.

Quality of Care: To determine if the MHS meets or exceeds benchmarks for health care 

quality as defined in HA and military medical department policies and guidance, and 

TRICARE contract specifications, with a particular focus on how the MHS performs relative 

to known national benchmarks. 

Patient Safety: To determine if the MHS has created a culture of safety with effective 

processes for ensuring safe and reliable care of beneficiaries.

Objectives

1. Assess prior recommendations and findings from relevant internal and external reports, 

including the last 10 years of Government Accountability Office and DoD Inspector 

General reports, to determine identified problems, actions taken to remedy the problems, 

and whether the remedy has been sustained. 

2. Review all relevant DoD, Military Department, and TRICARE policy standards and 

assess the degree to which the policies have been implemented. 

3. Evaluate data to assess compliance with existing policy or national standards. Determine 

how the MHS can consistently exceed these standards. Determine if any variance from 

the standards is due to data inaccuracy or inconsistency.

4. Review education and training documentation of health care professionals and staff 

regarding the execution of policies and assess knowledge of existing standards. 

5. Compare MHS performance to at least three civilian health systems, where standards are 

relevant and comparable.

6. Assess the experiences and perceptions of MHS patients regarding access, quality, and 

safety standards.

7. Determine the effectiveness of governance in policy and system performance.

8. Identify current resources for access, safety, and quality efforts to the extent possible.

Methods

A three-pronged approach was used to meet the goals and objectives listed above, organized 

around the themes of access, quality, and safety. In the first phase, relevant DoD instructions 

and Service policies were reviewed and internal and external reports were assessed to understand 

findings and recommendations made (see Appendix 1.3 for a list of documents reviewed). 

Additionally, enterprise-wide data were collected and metrics examined to determine the extent 

to which they align with DoD and Service policies to inform MHS leadership’s understanding of 

MHS quality and performance. Details on the access, quality, and patient safety metrics used in 

the review can be found in corresponding report sections. The selected metrics were generally 

readily available from the military Services or DHA, applicable to defined data sources and 

14
Military Health System Review – Final Report August 29, 2014

validated algorithms, consistent with national standards (where such standards exist), and in use 

by leading health care systems. The most recent annual data were analyzed, including the 

current year, when available, and trends for recent years were evaluated where possible.

Second, site visits of a cross section of military treatment facilities (MTFs)7 were conducted with 

a primary goal of providing local validation of centrally collected data. An additional goal 

included review of potential gaps between processes and policy. The facilities chosen represent 

all three Services (Army, Navy, Air Force) and the National Capital Region Medical Directorate 

(NCR MD), are of varying size and scope, and have broad geographic distribution, including sites 

within and outside the continental United States (see Table 1.1). Site-specific data requests 

included the results of the last two Joint Commission surveys, any other regulatory site visit 

findings from the last three years, results of internal or external audits, the last two command 

inspection reports and command climate surveys, materials related to goals or strategic plans and 

relevant appointment access guidelines. (See Appendix 1.4 for site visit methodology.) 

The site visit assessment team consisted of senior Service Flag/General Officers; senior enlisted 

leaders; subject matter experts for access, quality, and safety representing each Service; and 

contract support personnel. This team was kept largely intact for all facility visits to provide 

consistent assessments across all sites. Survey questionnaires were sent to regional Service 

medical leadership and presented to facility leadership, subject matter experts, health care staff, 

and patients for independent feedback. Team members met with local MTF leadership and 

subject matter experts at the sites and conducted walking rounds in order to interview health care 

staff and patients. Finally, an independent contractor conducted town hall meetings with MTF 

staff and beneficiaries with the intent to allow unfiltered feedback on facility performance with 

regard to access, quality, and safety (see Appendix 1.5 for town hall meeting summaries and 

quantitative analysis). In addition, a website was made available to the public to allow for 

additional comments to be submitted via e-mail (see Appendix 1.6). These anecdotal comments 

were analyzed for the report. Overall, the submitted comments mirrored the themes in the town 

hall meetings. Site visit reports were created with qualitative and quantitative assessments of 

facility specific performance. The site visit team also provided summaries with operational 

feedback to the MTFs so the facility could improve and/or sustain any of the areas that were 

observed during the visit. 

7 The term “military treatment facilities (MTFs)” refers to the medical facilities of the Defense Health Agency and 

the Departments of the Army, Navy, and Air Force, including: academic medical centers (e.g., Walter Reed National 

Military Medical Center, Bethesda; San Antonio Military Medical Center); military community hospitals; and 

military clinics.

15

 
August 29, 2014 1. Introduction

Table 1.1 MHS Review Site Visit Locations

Locations

10th Medical Group, Air Force Academy, Colorado Springs, Colorado

48th Medical Group, RAF Lakenheath Air Force Base, Lakenheath, England

Fort Belvoir Community Hospital, Fort Belvoir, Virginia

Madigan Army Medical Center, Joint Base Lewis-McChord, Washington

Naval Health Clinic Patuxent River, Naval Air Station Patuxent River, Maryland

Naval Medical Center San Diego, San Diego, California

Winn Army Community Hospital, Fort Stewart, Georgia

Source: MHS Review Group, July 2014

Third, three high-performing civilian health care systems—Geisinger Health System, 

Intermountain Healthcare, and Kaiser Permanente —were selected for the purpose of comparing 

the MHS against health systems with similar structure (providers and health plan), size and scope 

of care. The comparison was based on relevant and available metrics on access, quality and 

safety. (See Appendix 1.7 for table of comparison metrics.)

1. Geisinger Health System is an integrated health services organization widely recognized 

for its innovative use of the electronic health record, and the development and 

implementation of innovative care models including ProvenHealth Navigator, an 

advanced medical home model, and ProvenCare program. The system serves more than 

3 million residents throughout 44 counties in central and northeastern Pennsylvania.8

Geisinger Health System includes 11 hospitals (total 1,638 beds), 78 community practice 

and specialty clinics, 1,683 licensed providers, and 4,370 nurses. Its most recent 

operating budget was $3.4 billion.

2. Intermountain Healthcare is a nonprofit system of 22 hospitals, a Medical Group with 

more than 185 physician clinics, and an affiliated health insurance company, 

SelectHealth. Its 33,000 employees serve 660,000 patients and plan members in Utah 

and southeastern Idaho. Its most recent operating budget was $5.4 billion.

3. Kaiser Permanente, founded in 1945, is one of the nation’s largest not-for-profit health 

plans, serving approximately 9.3 million members, with headquarters in Oakland, 

California. It includes providers and a health plan, offering health care in seven U.S. 

regions through 38 hospitals, 618 medical offices, 17,425 physicians, and 48,701 nurses. 

Its most recent operating budget was $53.1 billion.

To meet the requirements in the Terms of Reference for independent review, external experts 

were brought in to review the methodology and findings. The external experts are nationally 

8 Geisinger website: http://www.geisinger.org/about/.

16

 
Military Health System Review – Final Report August 29, 2014

renowned leaders in health care with a wide breadth of education and training. Their collective 

portfolio includes experience and recognized success with large-scale quality and safety 

analyses, clinical quality improvement, strategy, and health care informatics. Experts were asked 

to review either methodology or findings. Originally, the MHS planned to include the external 

methodology experts during early stages of the review. However, because of the timeline, hiring 

process, and individual availability, their contributions were limited to a post-hoc evaluation of 

the methodology. The external experts tasked with reviewing the findings were available to the 

work groups to discuss the methodology as well as the findings and also were given access to the 

report of the MHS Review Group. (See Appendix 6.2 for the reports from the external experts.) 

Data Analysis and Limitations

Data quality and analysis is a critical component of this review. Robust data analysis includes 

considerations of the following: variations in how a metric is defined; population demographics 

such as age and gender distribution; burden of illness in the population; environmental and 

geographic issues; and other factors influencing analyses and results. Even well-defined metrics 

may be substantially influenced by factors beyond the control of the medical facility. Without a 

context, isolated metrics and even groups of metrics may be improperly interpreted, resulting in 

erroneous conclusions, especially when comparing and ranking performance. External 

comparisons are often available for established standards or benchmarks of performance that are 

clearly defined and nationally recognized. Yet, identifying the most appropriate benchmark(s) 

and appropriately interpreting results given variable local context remains a challenge for many 

experts.

From a systems perspective, the emphasis in the civilian sector is increasingly on the use of data 

to drive decision-making by identifying areas of variation and opportunities for improving 

consistency. In the areas of patient safety, quality, and access, this paradigm revolves around 

leadership’s use of analytics to drive process improvement. While significant gains have been 

made in this arena across U.S. health care, coordinated efforts are hindered by a lack of 

comparative data and accepted benchmarks, particularly in the areas of patient safety and access. 

The three Services and DHA have varying degrees of analytical capability and resources. In 

conducting this review, personnel were assembled ad hoc to address the identified questions. 

The human resources required to perform a “deep dive” into external data related to access, 

quality, and safety is enormous and costly. In both the analysis of the performance of the MHS 

for the agreed upon measures, and in the attempt to compare MHS performance against other 

systems, a significant gap has been identified in the MHS; that is, analytic capability and 

capacity for systematically and routinely assessing quality and patient safety. 

Comparing and contrasting different health care systems is a challenging endeavor. It requires 

detailed understanding and knowledge of the context and practices of the involved organizations. 

Even with defined metrics, there are differences in how data are collected and aggregated. If 

these challenges are surmounted one is still left with the lack of a standardized data set in a 

health system to globally measure performance. Seemingly identical or similar measures are 

useful as general indicators of performance, but have limited utility in ranking systems. While it 

17
August 29, 2014 1. Introduction

may be possible, with considerable work, to compare a valid set of metrics between two health 

systems, that does not translate into the ability to determine which health system has better 

quality; it only identifies which of the two systems performs better against that specific measure. 

In support of this review, three external civilian organizations provided the MHS Review Group 

with access, quality of care, and patient safety data in an attempt to compare those organizations 

against the MHS. The effort to complete this task demonstrates the inherent challenges of 

comparing complex health care systems. Despite tremendous internal work and coordination, 

plus the willingness, cooperation, support, and effort of external institutions, the degree of data 

comparability was notably limited due to variations among definitions, processes, and objectives 

of the institutions. This situation illustrates the potential benefits of increased standardization 

throughout the health care community. Assessing and improving health care systems requires 

sufficiently robust, integrated, and coordinated information systems; a broad range of 

professional knowledge, skills, capabilities; and supportive organizational structures.

(See Appendix 1.8 for data analytics summary.) 

Organization of the Report

Following this section is an overview of the MHS and its governance, which provides the 

background context for the review. Next are focused assessments of performance in the MHS 

with regard to access, quality, and safety, and recommendations related to those findings. 

Detailed information regarding methodology and data related to each area of performance can be 

found in the appendix material. In considering the findings and recommendations of this report, 

important contextual background is necessary. The fundamental mission of the MHS, providing 

medical support to military operations, is different from that of any other health system in the 

United States. 

18
Military Health System Review – Final Report August 29, 2014

2. OVERVIEW OF DOD’S MILITARY HEALTH SYSTEM

Introduction

The Military Health System (MHS) provides a continuum of health services from austere 

operational environments through remote, fixed medical treatment facilities to major tertiary care 

medical centers distributed across the United States. Mission-critical aspects of the MHS include 

the ability to sustain an interdependent and self-supporting, responsive health care team. Force 

Health Protection is the critical support function of the MHS in providing a worldwide 

deployable defense force. 

The MHS combines health care resources from both the direct and purchased care components to 

provide access to high-quality health care for the 9.6 million beneficiaries, including Service 

members of the seven uniformed services, National Guard and Reserve members, retirees and 

their eligible family members, survivors, certain former spouses, and other individuals, while 

maintaining the capability to support military operations worldwide (see Table 2.1). The 

percentage of beneficiaries using MHS services increased from 83.3 percent in Fiscal Year 2011 

to 84.9 percent in Fiscal Year 2013.9

Where available and as space allows, eligible beneficiaries may obtain health care from military 

hospitals and clinics, referred to as military treatment facilities (MTFs), or from civilian 

providers. In Fiscal Year 2013, the MTFs included 56 hospitals, 361 ambulatory care clinics, 

and 249 dental clinics operating worldwide and employing 60,389 civilians and 86,051 military 

personnel.10

Table 2.1 Average Weekly Statistics in the MHS

MHS Statistics

20,000 inpatient admissions (5,000 direct care; 15,000 purchased care)

1.9 million outpatient visits (834,000 direct care; 1,042,000 purchased care)

2,288 births (943 direct care; 1,345 purchased care)

3.8 million health care claims processed

2.54 million prescriptions filled (926,554 direct care; 1.24 million retail pharmacies; 363,000 home 

delivery)

343,000 behavioral health outpatient services (61,000 direct care; 282,000 purchased care)

177,000 emergency room visits (28,000 direct care; 149,000 purchased care)

2014 MHS Review Group

Source: Military Health System Mart (M2), January 2014.

9 DoD Evaluation of the TRICARE Program: Fiscal Year 2014 Report to Congress. Accessed: July 28, 2014.

Available at: 

http://www.tricare.mil/tma/congressionalinformation/downloads/TRICARE%20Program%20Effectiveness%20(FY

%202014)%201.pdf.

10 Ibid.

19

 
August 29, 2014 2. Overview of DoD’s Military Health System

Roughly half of health care in the MHS is provided in the direct care component at MTFs by 

uniformed service personnel, DoD civilians, and/or contracted civilian health care 

professionals.11 Supplementing the direct care component, the purchased care component of 

TRICARE is composed of TRICARE-authorized civilian health care professionals, institutions, 

pharmacies, and suppliers who have generally entered into a network participation agreement 

with a TRICARE regional contractor. For Fiscal Year 2013, the purchased care network 

included 3,310 acute care hospitals and approximately 478,000 participating providers. Nonnetwork care is delivered by TRICARE-authorized providers who may choose to “participate” in 

TRICARE on a claim-by-claim basis.

The purchased care component of TRICARE includes:

• TRICARE North Region administered by Health Net Federal Services 

• TRICARE South Region administered by Humana Military 

• TRICARE West Region administered by United Health Care Military and Veterans

• TRICARE Overseas administered by International SOS

The TRICARE Overseas Program (TOP) is DoD’s health care program that provides health care 

support services to approximately 458,000 beneficiaries outside of the 50 States and the District 

of Columbia. Using discretionary authority, and recognizing the cultural differences in accessing 

care in host nation countries, the TOP contract requires the contractor to make its best effort to 

ensure that the TRICARE standards for access—in terms of beneficiary travel time, local 

community standards, appointment wait time, and office wait time for various categories of 

services contained in 32 C.F.R. § 199.17—are met for TOP Prime enrollees. Similar to the 

purchased care program in the 50 States and the District of Columbia, TOP is administered on a 

regional basis by the TOP Program Office and TRICARE Area Offices (Eurasia-Africa, Pacific,

and Latin America/Canada), supported by a health care support contractor. 

TRICARE offers beneficiaries a family of health plans, based on three primary options:

• TRICARE Prime is a health benefit similar to a health maintenance organization, offered 

in many areas. Each enrollee chooses or is assigned a primary care manager, a health 

care professional who is responsible for helping the patient manage his or her care, 

promoting preventive health services (e.g., routine exams, immunizations), and arranging 

for specialty care provider services as appropriate. Access standards apply to waiting 

times to get an appointment and waiting times in doctors’ offices. A point-of-service 

option permits enrollees to seek care from providers other than their assigned primary 

care manager without a referral, but with significantly higher deductibles and cost shares 

than those under TRICARE Standard. Of the 9.6 million eligible DoD beneficiaries, 5.3 

million use TRICARE Prime.

11 Ibid.

20

 
Military Health System Review – Final Report August 29, 2014

• TRICARE Standard is the non-network benefit, formerly known as the Civilian Health 

and Medical Program of the Uniformed Services (CHAMPUS), open to all eligible DoD 

beneficiaries, except active duty Service members. Beneficiaries who are eligible for 

Medicare Part B are also covered by TRICARE Standard for any services covered by 

TRICARE but not covered by Medicare. An annual deductible (individual or family) and 

cost shares are required.

• TRICARE Extra is the network benefit for beneficiaries eligible for TRICARE Standard. 

When non-enrolled beneficiaries obtain services from TRICARE network professionals, 

hospitals, and suppliers, they pay the same deductible as TRICARE Standard; however, 

TRICARE Extra cost shares are reduced by 5 percent. TRICARE Extra is not available 

overseas.

In addition to TRICARE benefits, some military families also rely on state and other federal 

programs (e.g., Medicaid, Medicare) to meet specific needs. For some “dual-eligible” 

beneficiaries, such programs provide home care, disposable supplies, respite care, and equipment 

that augment TRICARE. For “TRICARE-for-Life” beneficiaries – i.e., those eligible for 

Medicare who have both Medicare Parts A and B – TRICARE functions generally as a 

supplement to Medicare. Other non-medical programs available from the Military Departments 

and community programs provide additional resources for children connected to military 

families. 

Beneficiary Demographics

TRICARE beneficiaries consist of two distinct populations: sponsors and dependents. Sponsors 

are typically active duty Service members, National Guard/Reserve members, or retired Service 

members. Thus, the sponsor is the person who is serving or who has served on active duty or in 

the National Guard or Reserves. “Dependent” is defined in 10 U.S.C. § 1072 and includes a 

variety of relationships, for example, spouses, children, and certain former spouses who have not 

remarried. 

The Army has the most beneficiaries eligible for Uniformed Services health care benefits, 

followed (in order) by the Air Force and the Navy (including the Marine Corps)

12 (see Table 

2.2). Although retirees and their family members constitute the largest percentage of the eligible 

population (56 percent) in the United States, active duty personnel (including Guard/Reserve 

Component members on active duty for at least 30 days) and their family members make up the 

largest percentage (66 percent) of the eligible population abroad.13 Mirroring trends in the 

civilian population, the MHS is confronted with an aging beneficiary population, with roughly 

22 percent of beneficiaries over age 65 and an additional 22 percent between the ages of 45 and 

12 Ibid. 13 Ibid.

21

 
August 29, 2014 2. Overview of DoD’s Military Health System

64 in Fiscal Year 2013.14 There is a roughly even distribution of beneficiaries by sex; 4.88 

million male and 4.70 million female.

Table 2.2 Distribution of Beneficiaries across the Military Departments and National Capital 

Region Medical Directorate

Military Department/Agency Army

Navy

(including 

Marine Corps)

Air Force National Capital 

Region Medical 

Directorate

Number of beneficiaries 3.9 million 2.81 million 2.61 million 500,000

Number of health care 

personnel (total/active duty)

95,000/45,000 63,000/40,000 61,000/33,000 10,823/4,494

Number of medical centers 8 2 3 1

Number of hospitals 14 16 10 1

Number of clinics 107 primary 

care

107 62 primary 

care

3

Appropriations (Fiscal Year 

2013)

$11.7 billion $9.59 billion $5.9 billion $1.3 billion

Source: 2014 MHS Review Group; DoD Evaluation of the TRICARE Program: Fiscal Year 2014 Report to Congress, 

July 2014. Available at: 

http://www.tricare.mil/tma/congressionalinformation/downloads/TRICARE%20Program%20Effectiveness%20(FY%20

2014)%201.pdf.

Policies for Priority of Access

Active duty personnel receive top priority access and are entitled to health care in a MTF in 

accordance with 10 U.S.C. § 1074. Dependents of active duty personnel are “entitled, upon 

request, to medical and dental care” on a space-available basis at a military medical facility (10 

U.S.C. §1076). Further, 10 U.S.C. § 1074 states that “a member or former member of the 

uniformed services who is entitled to retired or retainer pay may, upon request, be given medical 

and dental care in any facility of the uniformed service” on a space-available basis. Thus, since 

1958 priority has been given to active duty Service members and their dependents in receiving 

medical and dental care at any facility of the uniformed services over military members who are 

entitled to receive retired pay and their dependents (Public Law No. 85-861). Subsequent 

enactments gave priority to TRICARE Prime beneficiaries over TRICARE Standard 

beneficiaries.

DoD Military Medical Operations

Although not within the scope of this report, it is important to acknowledge that over the past 13 

years of military operations, the one constant in the MHS is the importance of health care 

delivery across the full spectrum of operations in service to our warriors. As of May 2014, there 

14 Ibid.

22

 
Military Health System Review – Final Report August 29, 2014

are nearly 1.4 million current members of the Armed Forces. Since the onset of the conflicts in 

Afghanistan and Iraq, until the end of 2013, 2.6 million troops have been deployed, all of who 

had to be medically ready, and uniformed medical personnel have deployed in support of combat 

operations on a continuous basis. Military medicine has achieved unprecedented outcomes 

despite the lethality of modern combat; the MHS has treated more than 52,000 Service members 

wounded in action,15 and the total number of Service members killed in action is just more than 

5,300, for a died of wounds rate of approximately 10 percent, the lowest in history.

The overall mortality rates are decreasing as Service members reach higher levels of medical 

care in a shorter period of time and thus survive longer after injury. Improvements in rapid 

evacuation from the battlefield and transport through the echelons of care—forward surgical 

team to combat surgical hospitals to regional medical centers to the continental United States 

(see Figure 2.1)—have resulted from advances in medical training of and equipment available to 

first responders and nonmedical personnel and from dramatic improvements in availability and 

capability of forward-deployed advanced surgical and critical care facilities. In the operational 

environment, military medicine has remained at the forefront of innovation while sustaining 

health care delivery practices to achieve desired clinical outcomes while promoting patient 

safety. Although deployed Forces are employed in remote and/or austere environments, the 

delivery of health care adheres to evidence-based, outcome-oriented management principles, 

which is overwhelmingly illustrated by the survival rate of our warriors on the battlefield. 

Military medicine often leads in health care innovation and delivery, particularly in times of 

conflict; recent examples include advances in amputee care and en route critical care. However, 

as a comprehensive health system, it is influenced by, and must be responsive to, improvements 

in the civilian health care sector. While the emphasis in medicine must be on the personal 

interaction of the patient with his or her provider, the modern approach to the delivery of care 

requires an integrated perspective that incorporates a systems-based strategy to problem solving, 

while continuing to maintain an individualized approach to the patient. Although this report 

reviews patient satisfaction data, its main purpose is to evaluate the MHS as a system. This does 

not minimize the importance of each and every patient encounter; rather, it reflects the realities 

of time and scope. 

15 Source: http://www.defense.gov/news/casualty.pdf.

23

 
August 29, 2014 2. Overview of DoD’s Military Health System

Figure 2.1 MHS Global Distribution of MHS Direct Care Platform 

Source: 2014 MHS Review Group, July 2014

 

Overview of Military Health System (MHS) Governance16

The MHS can be described as a federated health care system with responsibility for the delivery 

of safe, high-quality care shared among the Office of the Assistant Secretary of Defense for 

Health Affairs (ASD(HA)), the Military Departments (Services), and the Defense Health Agency 

(DHA). This governance structure follows from DoD’s overall organizational structure, with the 

MHS nested within the Department (see Figure 2.2). As demonstrated in Figure 2.2, the 

ASD(HA) reports to the Under Secretary of Defense for Personnel and Readiness (USD[P&R]), 

who in turn reports to the Secretary of Defense. 

Army, Navy, and Air Force medical commands report through their Service Chiefs to their 

respective Military Department Secretary and then to the Secretary of Defense. The federated 

16 In this report, terms such as MHS Governance, MHS governing bodies, governance structure, 

governing committees, and the like refer to the DoD management officials with authority over 

components or functions of the MHS (e.g., ASD(HA), Secretaries of Military Departments, Surgeons 

General, Director, DHA) and the governance councils referred to in DoD Directive 5136.13 (para. 5.a(1)) 

that provide advice and assistance to those officials. 

24

 
Military Health System Review – Final Report August 29, 2014

approach with shared responsibilities can be seen all the way to the level of the MTFs. 

ASD(HA) manages the resources that fund the Service medical departments and thus the MTFs,

primarily through the Defense Health Program (DHP) appropriation, but the MTFs are run by

military commands, and therefore are under the direction and control of the Services. At the 

MTF level, the Services maintain responsibility to man, train, and equip those commands to meet 

mission requirements. At a high level, the MHS collaboratively develops a strategy to meet 

policy directives and targets, with the Service components and/or the DHA responsible for 

execution. 

In response to the interest in improved coordination of the MHS and in stronger alignment 

among the Services, other DoD components with a health mission, and ASD(HA), the Deputy 

Secretary of Defense chartered a MHS Governance Task Force in June 2011 to conduct a review 

of MHS governance. The Task Force was charged with providing options for governance of the 

MHS as a whole, of enhanced multi-service medical markets, and of the National Capital Region 

health system. The Task Force submitted its report in September 2011, resulting in a Deputy 

Secretary of Defense memorandum of March 2012 (“Implementation of Military Health System 

Governance Reform”), which provided direction for a new governance structure for the MHS. 

As part of that effort, the prior governance structure of integrating councils reporting to the 

Senior Military Medical Action Council (SMMAC) was replaced by the present structure, which 

separates policy from execution (see Figure 2.2). 

The Defense Health Agency (DHA) was established “... to assume responsibility for shared 

services, functions and activities of the MHS and other common clinical and business 

processes.”17 The Director of the DHA is required to carry out assigned functions in accordance 

with direction from the ASD(HA), “adopted with the advice and assistance of governance 

councils established by the USD(P&R) and ASD(HA), including senior representatives of the 

Military Departments.” This structure has been in place since October 2013 and has already 

demonstrated significant gains in communication and alignment among HA, the Services, and 

DHA. The approach taken emphasizes collaborative work at all levels with representation of the 

Joint Staff and all Service components, greater involvement of General Officers and Senior 

Executives in leading the effort, closer alignment of DHA and the Services, and leveraging of the 

Medical Deputies Action Group (MDAG; see Figure 2.2) to provide advice and assistance on 

tactical management of the MHS.

17 Deputy Secretary of Defense Memorandum. March 11, 2013. “Implementation of Military Health System 

Governance Reform.” Available at: 

http://www.ausn.org/Portals/0/pdfs/Deputy%20SECDEF%20Carter%20Memo%20on%20DHA%20-

MHS%20Transformation%203-2013.pdf.

25

 
August 29, 2014 2. Overview of DoD’s Military Health System

Figure 2.2 Organizational Structure of the Military Health System within the Department of Defense

Source: 2014 MHS Review Group, July 2014

The MDAG is chaired by the Principal Deputy Assistant Secretary of Defense for Health Affairs, 

with representatives from the three Service components, Joint Staff, and HA at the deputy level. 

It advises and assists in the active management of the system while respecting each entity’s 

authority and responsibilities, as illustrated in the following:

C2

ASD(HA) – Assistant Secretary of Defense for Health Affairs

BUMED – US Navy Bureau of Medicine and Surgery

CJCS – Chairman of the Joint Chiefs of Staff 

CMC – Commandant of the Marine Corps

CNO – Chief of Naval Operations

CSA – Chief of Staff, Army

CSAF – Chief of Staff, Air Force

DASD – Deputy Assistant Secretary of Defense

JSS – Joint Staff Surgeon

MAJCOM – Major Command, Air Force

MBOG –- Medical Business Operations Group 

MDAG – Medical Deputies Action Group

MEDCOM – United States Army Medical Command

MHS – Military Health System

MHSER – Military Health System Executive Review 

MOG – Medical Operations Group

MPOG – Manpower and Personnel Operations Group

MTF – Military Treatment Facility

NCR – National Capital Region

PAC – Policy Advisory Council 

PDASD – Principal Deputy Assistant Secretary of Defense

SG – Surgeon General

SMMAC – Senior Military Medical Action Council 

USD(P&R) – Under Secretary of Defense for Personnel and 

Readiness

USUHS – Uniformed Services University of the Health 

Sciences

Coordination & Assistance

LEGEND

ABBREVIATIONS

26
Military Health System Review – Final Report August 29, 2014

1) ASD(HA) serves as the principal advisor to the Secretary of Defense and USD(P&R) for 

all DoD health and force health protection policies, programs, and activities, and:

a. Ensures the effective execution of the DoD medical mission, providing and 

maintaining readiness for medical services and support to members of the Military 

Services, including during military operations;

b. Exercises authority, direction, and control over DoD medical personnel authorizations 

and policy, facilities, programs, funding and other resources in the DoD;

c. Serves as resource manager for all DoD health and medical financial and other 

resources; 

d. Prepares and submits, in the DoD Planning, Programming, Budgeting and Execution 

process, a DoD Unified Medical Program budget to provide resources for the DoD 

MHS;

e. May not direct a change in the structure of the chain of command within a Military

Department or with respect to medical personnel assigned to that command.

18

2) DHA is established as a Defense Agency, under the authority, direction, and control of 

the USD(P&R), through the ASD(HA) and:

a. Manages TRICARE;

b. Manages and executes the Defense Health Program appropriation and DoD MHS 

funding;

c. Exercises management responsibility for shared services, functions, and activities of 

the MHS and its common business and clinical processes; 

d. Supports the effective execution of the DoD medical mission;

e. Collaborates with the Military Departments to ensure an integrated and standardized 

TRICARE and health care delivery system19

3) Military Service Components are responsible for: 

a. Communicating Service-specific requirements and requests relating to shared 

services, activities, and functions to the Director, DHA;20

b. Ensuring that the Service medical departments remain accountable for the delivery of 

patient care, and related medical and health services in facilities under their 

jurisdiction, consistent with this directive;

21

c. Manning, training, and equipping of Service assets to meet mission requirements.22

18 DoD Directive 5136.01, September 30, 2013. Available at: 

http://www.dtic.mil/whs/directives/corres/pdf/513601p.pdf.

19 DoD Directive 5136.13, September 30, 2013. Available at: 

http://www.tricare.mil/tma/ams/downloads/513612p.pdf.

20 Ibid.

21 Ibid.

22 Title 10, United States Code Sections 3013, 5013, 8013.

27

 
August 29, 2014 2. Overview of DoD’s Military Health System

Figure 2.3 Governance Structure of the Military Health System

Source: 2014 MHS Review Group, July 2014

Figure 2.3 displays the governance structure within the MHS. The Military Health System 

Executive Review (MHSER) is comprised of senior-level DoD leadership and charged with 

providing input on strategic, transitional, and emerging issues in the MHS. It advises the Office 

of the Secretary of Defense and the Office of the Deputy Secretary of Defense on performance 

challenges and direction. It is chaired by the USD(P&R), and includes the Principal Deputy 

Under Secretary of Defense (Personnel and Readiness), the ASD(HA), the Service Vice Chiefs, 

Military Department Assistant Secretaries for Manpower and Reserve Affairs, the Assistant 

Commandant of the Marine Corps, the Director of Program Analysis and Evaluation, the 

Principal Deputy Under Secretary of Defense (Comptroller), the Director of the Joint Staff, and 

the Surgeons General (as ex-officio members).23

The SMMAC is chaired by the ASD(HA), and includes the Principal Deputy Assistant Secretary 

of Defense (Health Affairs), Military Department Surgeons General, DHA Director, Joint Staff 

Surgeon, and other attendees as required. The Council presents enterprise-level guidance and 

operational issues for decision-making by the ASD(HA). 

The MDAG reports to the Council, which ensures that actions are coordinated across the MHS 

and are in alignment with MHS strategy, policies, directives, and initiatives.

Four supporting governing bodies, consisting of Flag/General Officers from the Service medical 

departments and Senior Executives from DHA, report to the MDAG. Each group has specific 

roles, but all are focused on sustaining and improving the MHS. 

23 MHSER Charter. March 17, 2004. Available at: 

ftp://98.130.80.93/HEALTH%20AFFAIRS%20DOD/DOD%20Academic%20Strategy/CASSCELLS%20PAPERS/

LIVE_LINK/MHSER%20Charter.pdf

28

 
Military Health System Review – Final Report August 29, 2014

• The Medical Operations Group (MOG) carries out assigned tasks and provides 

enterprise-wide oversight of the direct and purchased care systems. 

• The Medical Business Operations Group (MBOG) provides a forum for providing 

resource management input on direct and purchased care issues. 

• The Manpower and Personnel Operations Workgroup (MPOG) supports centralized, 

coordinated policy execution and guidance for development of coordinated human 

resources and manpower policies and procedures for the MHS. 

• The Enhanced Multi-Service Markets (eMSM) Leadership Group provides a forum for 

managers of geographic MHS markets to discuss clinical and business issues, policies, 

performance standards, and opportunities. 

Finally, the ASD(HA) is supported and advised by the Policy Advisory Council (PAC), 

composed of the Deputy Assistant Secretary of Defense (Health Affairs), the DHA Deputy 

Director, the Deputy Surgeons General, and a representative of the Joint Staff. The PAC 

provides a forum for supporting MHS-wide policy development and oversight in a unified 

manner. 

The MTFs, as military commands, are controlled and operated by their respective Military 

Departments (Army, Navy, and Air Force). Two notable exceptions are Walter Reed National 

Military Medical Center and Fort Belvoir Community Hospital, which report to DHA and are not 

military commands. This configuration is the result of MHS governance reform. Coincident 

with establishment of DHA, the former Joint Task Force National Capital Region Medical 

transitioned to become the National Capital Region (NCR) Medical Directorate. NCR Medical 

Directorate (NCR MD) has oversight and responsibility for execution of patient safety and 

quality programs for the two NCR MD medical facilities. 

Governance Reform Related to Performance Improvement

As described above, the major changes in MHS governance are the governance bodies (MHSER, 

SMMAC, MDAG, MOG, MBOG, MPOG; see Figure 2.3) and the standup of DHA. DHA’s 

mission includes supporting greater integration of DoD’s direct and purchased health care

delivery systems in order to achieve better medical readiness, improved health, enhanced 

experience of care, and lower health care costs. It meets this mission through, among other 

activities, the shared services it provides in support of the MTFs, in the management of the 

Defense Health Program, and in providing an enterprise-wide view of the MHS for the 

governing bodies. 

As a result of these recent reforms, the MHS is in transition, and work remains to be done in 

clarifying roles and relationships among the components and in establishing mechanisms to 

monitor and drive performance as a system. Presently, the overarching goals of the MHS are 

captured in the Quadruple Aim (concept modified from the Triple Aim of the Institute for 

Healthcare Improvement):

29
August 29, 2014 2. Overview of DoD’s Military Health System

• Readiness (Goal – Increased Readiness): ensuring that the total military force is 

medically ready to deploy and that the medical force is ready to deliver health care 

anytime, anywhere

• Population Health (Goal – Better Health): keep people healthy and reduce the frequency 

of the visits to hospital and clinics

• Experience of Care (Goal – Better Care): provide a care experience that is safe, timely, 

effective, efficient, equitable, and patient- and family-centered

• Per Capita Cost (Goal – Lower Cost): create value by focusing on quality, eliminating 

waste, reducing unwarranted variation

While HA, DHA, and the Service medical departments all use the Quadruple Aim to define the 

overarching goals on which their respective strategies are based, there are differences in the 

specific metrics each Service uses to monitor performance in their respective MTFs, and in how 

measures are defined and how data are collected and reported. This results in challenges in 

assessing the performance of the MHS as an enterprise. Within MHS governance, the measures 

are reviewed by multiple committees, which all report to the MOG. (The MOG includes a 

synopsis of its discussion in its minutes, which are reviewed by the MDAG, whose minutes are 

in turn reviewed by the SMMAC.) 

Governance is beginning to address variability in metrics used, as is evidenced by the adoption 

of a single dashboard for the enhanced multi-service markets (eMSMs). However, the eMSMs 

represent only 30 to 40 percent of the MHS and their analytical emphasis is weighted toward 

business metrics, in keeping with the eMSM focus on management of the market. The MDAG 

performs quarterly performance reviews of the eMSMs, and a summary of those reviews is then 

presented to the SMMAC. At the time of this report there were no scheduled reviews by the 

MHSER of specific MHS quality, patient safety, or access measures. 

DHA and the Army, Navy, and Air Force medical departments shape performance improvement 

goals and align them to their respective strategic plans (see Appendix 2.1 for DHA and Service 

Performance Improvement activities). Each Service uses a variety of tools and methods 

throughout the medical department to the level of the MTF. Lean and Six Sigma® are the 

common frameworks and methodology used for performance improvement. Each of the 

Services uses a Service-specific web-based portfolio software management system for program 

management, tracking, and reporting. 

The DHA Director has the authority and responsibility to oversee Patient Safety and Quality, as 

outlined in DoD policy (DoDM 6025.13 and JTF 6025.01 Quality Manual). The Chief Medical 

Officer, who is also the Director of Health Care Operations in DHA, has responsibility for the 

programs and offices that support MHS enterprise efforts for improving patient safety and health 

care quality. Examples of the collaborative efforts in process improvement for patient safety and 

quality are illustrated below.

The DoD Patient Safety Program (PSP) is established under the Clinical Support Division, 

DHA. The DoD PSP manages its operations through the Patient Safety Improvement 

Collaborative (PSIC); PSIC is chaired by the PSP Director and includes representatives from 

30
Military Health System Review – Final Report August 29, 2014

the three Services, NCR MD, Uniformed Services University, and the TRICARE Regional 

Offices (TROs). The Patient Safety Analysis Center (PSAC), within the Clinical Support 

Division, collects, maintains, analyzes, and submits reports on patient safety performance 

metrics submitted from the DoD MTFs. The PSIC supports the work of the MHS Clinical 

Quality Forum. 

The Clinical Quality Forum (CQF) is a collaborative group with representation from all 

components of direct and purchased care. It has the responsibility to assess clinical quality 

across the MHS. CQF assessments are based on relevant clinical performance indicators for 

health care system performance, including beneficiary and stakeholder perceptions of care 

and activities focused on quality assurance, patient safety, and risk management events. 

Function-focused working groups and advisory panels under CQF provide insight, 

recommendations, and activities to enhance clinical quality and safety.

Additionally, the CQF develops, or may endorse, recommendations for clinical quality 

improvement for approval by the MOG and/or the MDAG. Each of the Services implements 

quality improvement efforts at MTFs throughout the world in accord with their 

organizational structure through Service leadership.

The new governance council structure has facilitated coordination among the Services. It is 

expected that this will translate into the Services coming to agreement in adopting a single 

approach in common areas. Examples of where this has already happened include the eMSM 

dashboard metrics and the patient-centered medical home model. 

Component Responsibilities within the Military Health System

In sum, HA is responsible for policy, and the Services for execution. The appropriate level 

governance committee develops the recommended strategy to meet goals and the metrics used to 

measure progress toward those goals. If consensus cannot be reached, the issue is elevated to the 

next level of governance council for consideration, with the expectation that the MHSER is the 

final MHS forum for issues that cannot be resolved at a lower level. DHA supports governance 

by monitoring the performance of the enterprise and by providing analyses as requested to a 

governing body. 

Many of the issues identified in this report are generated by differences among the MHS 

components in the data collected to monitor performance and make decisions. A common set of 

metrics, with targets that would roll up from the MTFs, through the Services and NCR MD, to

MHS and DoD leadership, to include the MHSER and the Military Department Secretaries, 

would improve MHS’s ability to identify variance and track performance as a system. Because 

measures and metrics are aspects of execution and implementation, the Services, through 

governance, are responsible for proposing a common set of measures to HA and meeting policy 

intent and direction. In this endeavor, DHA plays a supporting role. Once the metrics have been 

approved by HA, the Services and the MHS governing committees would conduct performance 

reviews. 

31
August 29, 2014 2. Overview of DoD’s Military Health System

Similarly, ideally, the Services through governance would generate a strategic plan to meet 

policy direction, which would then be reviewed and approved by HA, which has an oversight 

role to ensure that the plan meets the intent of policy. Again, DHA would support this effort 

with data analysis and administrative support when asked. This is an improvement over past 

governance, under which the entities responsible for execution of the strategic plan were not 

responsible for generating that plan. 

Resource Support for Patient Quality, Access and Safety

Using Fiscal Year 2013 as a representative year, an analysis was performed to identify MHS 

resources primarily supporting patient quality, access, and safety. The results indicate that 

substantial personnel and funding are devoted to these important functions system wide. In 

aggregate, more than 9,800 full-time-equivalent staff and more than $875 million per year are 

reported as directly supporting quality, access, and safety efforts. Most of these resources fall 

within the In House Care Budget Activity Group, which accounts for resources supporting care 

within the MTFs. Based on FY 2013 actual values, it is estimated that up to 8 percent of In 

House Care staff—to include military, civilian, and contractors—and up to 11 percent of In 

House Care financial resources directly support quality, access, and safety initiatives. 

Conclusions 

The MHS is a unique global health care organization with multiple missions and layers of 

complexity. Oversight and governance are, by nature, complex given the missions of the 

Military Departments and DoD’s obligations to provide health care for active duty Service 

members and their families, as well as retirees and their families. The following sections provide 

assessments of the MHS’s performance with regard to access to care, quality of care, and patient 

safety, measured against its own metrics, as compared to other high-performing health care 

systems, and compared to national benchmarks. Findings identify successes as well as 

opportunities for improvement, combined with actionable recommendations.

32
Military Health System Review – Final Report August 29, 2014

3. ACCESS TO CARE IN THE MILITARY HEALTH SYSTEM

Introduction

Access to care is defined as “the timely use of personal health services to achieve the best health 

outcomes.”24 Access to care is influenced by many factors, including community health care 

resources, insurance coverage, financial status, proximity to care, and technology. Timely access 

to health care is a universal concept applicable to all health systems; however, the definitions and 

measures of timeliness are not standardized nationally. 

Unique to the MHS, access standards are identified in Section 199.17(p)(5) of 32 Code of Federal 

Regulations (32 C.F.R. § 199.17(p)(5)); see Appendix 3.1). These standards include:

• 30-minute drive time for primary care 

• Specialty care appointments within four weeks

• Routine appointments within one week

• Urgent care appointments generally not to exceed 24 hours

• Emergency room access available 24hrs/7 days per week

• 60-minute drive time for specialty care 

• Office wait times should not exceed 30 minutes unless emergency care is being rendered 

to another patient

MHS enrollees to Medical Treatment Facilities (MTFs) have options for accessing care, 

including MTF appointments, Secure Messaging, TRICARE Online booking, or referrals to the 

private sector if the MTF does not have capability or cannot meet the MHS access standards. 

The Nurse Advice Line enhances access for all MHS beneficiaries within the continental United 

States. 

In the MHS, the direct care and purchased care components operate in tandem to meet the 32 

C.F.R. § 199.17 standards for the 5.3 million enrolled TRICARE Prime beneficiaries. The 

purchased care component is a safety net to ensure that Prime beneficiaries have an avenue for 

care when the direct care component cannot provide it. Annually, direct care provides an 

average of 43.4 million office visits while the MHS spends more than $15 billion annually in 

purchased care. 

The following summary was compiled by subject matter experts from each of the Services and 

from the Defense Health Agency (DHA), which also represents the National Capital Regional 

Medical Directorate (NCR MD). It includes a review of access governance in the MHS, an 

24 Institute of Medicine, Committee on Monitoring Access to Personal Health Care Services (1993). Access to 

Health Care in America. Washington, D.C.: National Academies Press.

33

 
August 29, 2014 3. Access to Care in the Military Health System

examination of policy gaps, a review of current access performance, analysis of seven site visits, 

and findings and recommendations. 

Access to Care Governance 

DHA is responsible for oversight of the purchased care component and regularly reviews 

contract performance for care delivered in the purchased care component, including available 

access data. In the direct care component, Service- and DHA-level guidance sets the conditions 

for MTF leadership to build and sustain a culture of continuous process improvement to ensure 

their access to care program is meeting standards set by DoD policy. Site visitors noted minor 

variation in governance on this issue among the seven MTFs. The Air Force has specific Group 

Practice Managers who are responsible for monitoring and managing access. The Army, Navy,

and DHA generally have assigned MTF Access Managers.25 MTF and clinic leaders are aware 

of their access performance and challenges, primarily through excellent communication with the 

access managers. Two of the seven sites had a multidisciplinary access to care group or 

committee for collaborative discussion and sharing of best practices to facilitate an 

organizational and strategic approach to access. 

Ultimately, the commanding officer or director of each facility is responsible for access within 

the MTF, which is accomplished through performance monitoring reviews. Across Services and 

DHA, the total number of metrics used in access reviews varies; however, all review acute 

access in primary care, primary care manager (PCM) continuity, secure messaging enrollment, 

Nurse Advice Line calls/dispositions, and emergency department utilization. Information on 

access flows between the MTF commander, intermediate commands, headquarters, and the 

Service Surgeon General; however, access data are not consistently transmitted above the level 

of the Surgeons General across the Services. 

Primary care access for all three Services and DHA is reviewed by the Medical Operations 

Group (MOG) through the Tri-Service Patient Centered Medical Home (PCMH) Advisory 

Board’s quarterly update. The update includes data on PCM continuity, access metrics for same 

day and routine appointing, emergency department utilization, patient satisfaction, staff 

satisfaction, and quality metrics. Best practices are identified and shared across the Services and 

DHA. The MOG also receives monthly performance metrics on the recently launched Nurse 

Advice Line for CONUS beneficiaries.

 

In 2014, the MOG approved the formation of the MHS Access Improvement Work Group (MHS 

AIWG) to facilitate access standardization in primary and specialty care across the Services. 

The MHS AIWG is currently drafting a DoD Instruction to standardize access business rules at 

all MTFs (see AIWG charter in Appendix 3.2). The MOG also approved the formation of the 

Tri-Service Specialty Care Advisory Board to standardize specialty care product lines including 

25 One small multidisciplinary clinic had someone from each department managing that department’s 

access. 

34

 
Military Health System Review – Final Report August 29, 2014

business rules for access, templates, secure messaging, operating rooms, and performance review 

metrics. 

Policy Review and Identification of Gaps 

The overarching guidance on access to care is set by 32 C.F.R. § 199.17, which has been in place 

since 1995. Critical supporting guidance includes Office of the Assistant Secretary of Defense 

(Health Affairs) (ASD(HA)) instruction, “TRICARE Policy 11-005, TRICARE Policy for 

Access to Care” (HA Policy 11-005) and the TRICARE program manuals. The policy review 

includes assessment of vertical gaps between higher and lower echelons and horizontal gaps 

between the Services and the NCR MD. 

ASD(HA) – Overarching Guidelines

HA Policy 11-005 implements 32 C.F.R. § 199.17 and confirms the applicability of the 

guidance to overseas locations and to TRICARE Overseas Program (TOP) contractors. The 

policy also encourages the Services to use the MHS Guide to Access Success (2008) as

guidance. Although not “policy” explicitly, the Guide remains central to the management of 

access within the MHS. HA Policy 09-15 directs each Service to implement the PCMH 

model of primary care, which includes enhanced access to care. 

Military – Service-Level Instructions

Each of the Services and DHA has established policy letters, instructions, and directives for 

Service-specific guidance on meeting the access standards. (Summaries of DoD and Servicelevel policies and orders can be found in Appendix 3.3.) 

TRICARE – Purchased Care Guidelines

The purchased care component is expected to meet access standards, administered through 

the TRICARE program manuals. Although evaluation of compliance is defined in the 

contract, it does not require the same level of detail required in the direct care component. 

For example, patient satisfaction with length of time to appointment is used as a surrogate 

measure of primary care access in lieu of detailed access to care compliance data. 

External Reviews

A 10-year retrospective review of DoD Inspector General (IG) and Government 

Accountability Office (GAO) reports identified 34 potentially relevant documents, seven of 

which were related to access but only two of which were relevant to access for TRICARE 

Prime beneficiaries; however, neither addressed the 32 C.F.R. § 199.17 access standards. 

The remaining GAO reports focused on access to civilian providers for TRICARE Standard

beneficiaries, for whom access standards are not defined by DoD regulation since 

beneficiaries are free to use any authorized provider. (Information on the GAO reports is

found in Appendix 3.4.) 

35
August 29, 2014 3. Access to Care in the Military Health System

Vertical gaps are defined as inconsistencies between higher- and lower-level policies. One such 

gap was identified—neither the Services nor NCR MD have established a standardized 

methodology to assess the 30-minute office waiting times. This gap was also noted in the 

purchased care component. Review of Service and NCR MD policies found no significant 

horizontal gaps. 

Education and Training

The Services have invested in access to care training for relevant personnel at a variety of levels. 

Currently, 12 distinct educational opportunities provide training on access management. This 

training is designed to develop expertise among all roles and levels of MTF staff and facilitates 

common goals to:

1. Implement and sustain a systematic, proactive, programmatic, and responsive access 

management program 

2. Ensure all clinics/services meet or exceed the access standards 

3. Create an access curriculum for all levels of the organization 

Each of the Services has a course on access management. In addition, the MHS has 

implemented a Tri-Service Access Improvement Seminar. Given the redundancy of these 

courses, there is opportunity to standardize training across the Services. (Summaries of Servicelevel access courses are found in Appendix 3.5.)

The site visit team found that appointing staff members receive formal training on the Composite 

Health Care System (CHCS), which serves as the foundation for DoD’s electronic health record. 

Medical appointments in MTFs are scheduled in CHCS, based on locally developed appointment 

templates. In addition, some MTFs have clinic-specific orientation to train new staff on standard 

operating procedures, along with on-the-job training. CHCS training appears to be standardized 

across the seven MTFs visited by the MHS Review Group, but clinic-level training ranges from 

formal and documented training to undocumented, informal on-the-job training. While all seven 

facilities provide training for call center booking agents and front desk clerks, some MTFs have 

organization-wide customer relations training during initial orientation and only a few require 

annual customer service training. Clinic leadership at all sites expressed a desire to learn more 

about clinic operations, including effective management of access and demand forecasting. 

Given that customer service at every contact point in a health care facility can have a direct 

effect on the patient experience, there is an opportunity to standardize customer service training.

Methodology

The MHS Review Group evaluated multiple data sources to explore access performance for the 

purposes of this review. Chosen metrics met the following criteria:

1. Established, readily available, and understood across the organization to the highest 

degree possible 

2. Sufficient to assess compliance with 32 C.F.R. § 199.17

36
Military Health System Review – Final Report August 29, 2014

3. Relevant to access to care performance throughout the entire MHS, to include both the 

direct care and purchased care components

4. Include modes of enhanced access to virtual services provided to patients such as secure 

messaging or use of an online portal 

5. Include patients’ perception of and satisfaction with MHS access to care 

6. Limited to primary care and specialty care as a whole, rather than by product line, due to 

time constraints for the review 

7. Include on-site observations and interviews in order to validate other data sources

Measures to assess the direct care component were chosen based on ready availability through 

standard access reports housed at the TRICARE Operations Center (TOC) website. The TOC 

electronically collects appointment process details for all MTFs and provides MTF clinical staff 

and decision makers at all levels meaningful current and historical access reports using a DoD 

Common Access Card. Appointment processing and scheduling business rules built into CHCS 

are directly aligned with 32 C.F.R. § 199.17 and HA Policy 11-005. 

The following TOC data were reviewed at the MHS and Service levels, by facility type (Medical 

Center, Hospital, and Clinic), by location (Overseas or United States), and by facility name:

1. Average number of days to acute appointments

2. Average number of days to third next available acute appointment (primary care)

3. Average number of days to third next available routine appointment (primary care)

4. Average number of days to specialty appointments

5. Average number of days to third next available specialty appointment 

6. Percent of acute appointments meeting MHS access standards

7. Percent of specialty appointments meeting MHS access standards

The following metrics were used to assess enhanced access to care and satisfaction:

1. Percent appointments web-enabled for TRICARE Online (TOL) booking

2. Number of direct care enrollees in secure messaging

3. Percent of direct care enrollees registered in secure messaging who initiated contact with 

their PCM 

4. Number of calls to the Nurse Advice Line 

5. Percent of calls to the Nurse Advice Line by disposition

6. Satisfaction with Getting Care When Needed (Service Surveys)

7. Satisfaction with Access to Care (TROSS)

8. Satisfaction with Seeing a Provider When Needed (TROSS)

9. Satisfaction with Getting Care Quickly (HCSDB)

10. Satisfaction with Getting Care when Needed (HCSDB)

TRICARE regional contractors submit data on the purchased care component as part of the 

contract requirements. The current TRICARE contracts do not require the contractors to collect 

and report the same level of detail on access to care as is available in the direct care component. 

The contractors do not collect the same measures in the same way as the MHS direct care 

37
August 29, 2014 3. Access to Care in the Military Health System

components, and do not use the same information systems. Moreover, the overseas contract does 

not collect the same data as the U.S. contractors. Although comparisons between the direct and 

purchased care components are difficult, data were collected on the following metrics:

1. Number of enrollees per network provider - United States

2. Specialty care percentage within a 60-minute drive time - United States

3. Percent of appointments within 28 days - United States

4. Access to care composite (TRICARE Outpatient Satisfaction Survey [TROSS]) - United 

States

5. Seeing a provider when needed (TROSS) - United States

6. Getting care quickly composite (Health Care Survey for DoD Beneficiaries [HCSDB]) -

United States

7. Getting needed care composite (HCSDB) -United States

8. Overseas network satisfaction data

The analysis also incorporated a review of external benchmarks, site visit information, town hall 

meetings, and regional reviews. Given that there are no national benchmarks for access, the 

MHS Review Group assessed national health plan standards, which vary widely. The California 

State Department of Managed Health Care set specific timelines for non-emergent access in 

2002, which are used to benchmark the current MHS performance. (See access standards 

comparison in Appendix 3.6) 

Direct Care Component Analysis

The following section presents performance in access metrics for the direct care component. 

(See Appendix 3.7 for FY 2014 access measures displayed by facility, and Appendix 3.8 for the 

analysis of percent of appointments meeting MHS standards). The data apply only to the care 

delivered in the MTFs and has been split to show access within primary care and specialty care 

across the MHS and the Services by facility type and geographic location. The 32 C.F.R. § 

199.17 and California (CA) standards are displayed in Table 3.1.

Table 3.1 MHS vs. CA Standards Comparison

MHS Appointment Type MHS Standard CA Appointment Type CA Standard

Acute Generally within 24 hours Urgent 48 hours

Routine 7 calendar days Non-urgent primary care 10 business days

Specialty 28 calendar days Non-urgent specialist 15 business days

2014 MHS Review Group

Source: California Department of Managed Care, June 2014 

Acute Care

Average Number of Days to an Acute Appointment

Overall: As of April 2014, 88 percent of acute appointments planned in CHCS were delivered in 

primary care and 12 percent in specialty care. Thus, this metric is most applicable to primary 

38
Military Health System Review – Final Report August 29, 2014

care for the direct care component, but it does include both primary and specialty care acute 

appointments. In Fiscal Year 2014 to date, the average number of days to an acute appointment 

is 0.97 days overall, outperforming both the MHS and CA access standards (see Figure 3.1).

The median was 0.46 days and there were four minor outliers and seven major outliers.26 The 

four outliers with average days to acute care appointments greater than 1.18 days were 99th 

MDG- Federal O’Callaghan, 633rd Medical Group, NMC Portsmouth, and Ft. Belvoir 

Community Hospital. The seven major outliers with average days of 1.69 days or greater were 

35th Medical Group, Leonard Wood ACH, Irwin ACH, Darnall AMC, NH Guantanamo Bay, 

NHC Hawaii, and Walter Reed National Military Medical Center. (See Appendix Table 3.10-1 

and Appendix Figure 3.10-1.)

Figure 3.1 Average Number of Days to Acute Appointment –

Overall: MHS Access Standard < 1 Day

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

In general, the upward trend in the number of days to an acute care appointment reflects an 

increase in demand associated with the direct care component’s transition to the PCMH model of 

care and the move to enhanced access. Previously, 24-hour appointments were reserved for 

acute health issues; now, more same day appointments allow health issues previously seen on a 

routine (7-day) basis to be seen on a 24-hour basis. By standardizing appointment templates, the 

direct care component increased total primary care appointments by five percent between FY 

2011 and FY 2014. The direct care component also simplified appointment templates and 

26 Outliers for MHS-level data were calculated based on the interquartile range (IQR), which is the difference 

between the first and third quartiles of observed data. Minor and major outliers were defined as 1.5 times beyond 

IQR and 3 times beyond the IQR, respectively. 

0

1

2

3

Average Number of Days

Month-Year

DC Overall MHS Access Standard CA

39

 
August 29, 2014 3. Access to Care in the Military Health System

reduced the number of appointment types, which increased the number of acute appointments by 

35 percent. 

Because most staff rotations occur in the summer months, the average number of days to an 

acute care appointment is higher from May to August each year. This trend may have been 

further exacerbated in July and August 2013, which coincided with civilian employee furloughs. 

Civilian Comparison: Health System 3 reported an average days to acute care appointments of 

“less than one” day, which is consistent with the direct care component FY 2014 average 

performance of 0.97 days. Health System 3 reports that patients who have acute medical needs 

are able to be seen on a walk-in basis or are given an appointment for the same day. A recent 

Merritt Hawkins survey of 15 major U.S. metropolitan areas reported the time to see a family 

medicine provider of 18.5 days (the study did not specify acute or routine.)27 

Service Level: There is variation in performance. All Services currently perform better than the 

CA access standard with the Air Force (0.55 days) outperforming the MHS access standard. 

Army (1.07 days), Navy (1.17 days), and NCR MD (1.64 days) did not meet the MHS access 

standard in FY 2014 (see Figure 3.2).

Figure 3.2 Average Number of Days to Acute Appointment – By Service: MHS Access 

Standard ≤ 1 Day

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014

27 Merritt Hawkins 2014 Survey of Physician Appointment Wait Times and Medicaid and Medicare Acceptance 

Rates. (2014). Available at: 

http://www.merritthawkins.com/uploadedFiles/MerrittHawkings/Surveys/mha2014waitsurvPDF.pdf.

0

1

2

3

Average Number of Days

Month-Year

Air Force Army Navy NCR MD MHS Access Standard CA

40

 
Military Health System Review – Final Report August 29, 2014

Facility Type: There is variation in average days to acute appointments when data are broken 

out by facility type. The larger facility types show a higher average number of days to acute care 

appointments than hospitals and clinics. All three facility types performed better than the CA 

acute access standards (see Figure 3.3).

Figure 3.3 Average Number of Days to Acute Appointment – By Facility Type: MHS Access 

Standard ≤ 1 Day

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014

Location: Facilities located overseas performed better on average days to an acute care 

appointment than facilities located in the United States: 0.61 days in FY 2014 compared to 1.03 

days. Both groups performed better than the CA access standard. Overseas locations may 

perform better as a result of having a pre-screened population and higher staffing levels. (See 

Appendix Figure 3.9-1.)

Average Number of Days to Third Next Acute Appointment in Primary Care

MHS Level: The average number of days to third next appointment is a prospective health care 

industry standard measure and is considered an excellent measure of overall appointment 

availability. The Agency for Healthcare Research and Quality (AHRQ) and the Institute for 

Healthcare Improvement recommend measurement of average number of days to third next acute 

and routine appointments in primary care settings as a more sensitive measure of ATC. In FY 

2014 to date, the average number of days to the third next acute appointment is 1.86 days, down 

11 percent from 2.09 days in FY 2012 (see Figure 3.4). The MHS and CA acute access 

standards are included as goals to work toward, as there is no standard for average number of 

days to third next appointment. The overall range of observations is 0.44 days to 5.62 days with 

64 percent of MTFs performing better than the overall overage of 1.86 days. The median was 

0

1

2

3

Average Number of Days

Month-Year

Clinic Hospital Medical Center MHS Access Standard CA

41
August 29, 2014 3. Access to Care in the Military Health System

1.4 days and there were 6 minor outliers with average number of days to third next acute greater 

than 1.72 days (99th MDG- Federal O’Callaghan, 633rd Medical Group, 81st Medical Group, 

Darnall AMC, William Beaumont AMC, Walter Reed National Military Medical Center) and 

one major outlier with average number of days to third next acute greater than 3.44 days (60th 

Medical Group).28 (See Appendix Table 3.10-2 and Appendix Figure 3.10-2.).

Figure 3.4 Average Number of Days to Third Next Acute Appointment (Primary Care) –

Direct Care Component Overall

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014

Civilian Comparison: The average number of days to the third next acute appointment of 1.86 

days in direct care is better than Health System 2 and Health System 3, which averaged 9.02 

days and 11.63 days, respectively. 

28 Outliers for MHS-level data were calculated based on the interquartile range (IQR), which is the difference 

between the first and third quartiles of observed data. Minor and major outliers were defined as 1.5 times beyond 

IQR and 3 times beyond the IQR, respectively. See Appendix 3.10 for Outlier Analysis.

 -

 1

 2

 3

 4

Average Number of Days

Month-Year

DC Overall MHS Access Standard CA

42

 
Military Health System Review – Final Report August 29, 2014

Service Level: There is variation among the Services in the average number of days to the third 

next acute appointment (see Table 3.2 and Figure 3.5). To better understand these differences, 

the percent of 24-hour appointments was reviewed and appears to have an inverse correlation to 

third next acute measure. All of the Services have improved performance since FY 2012. 

Table 3.2 Average Number of Days to Third Next Acute Appointment (Primary Care) by Service

and Percent of 24-Hour Care Appointments

Navy Army Air Force NCR MD

Average Number of Days to Third Next Acute 

Appointment

1.00 1.91 2.24 2.53

% Primary Care Appointments Available for 24-Hour 

Care

54% 46% 36% 30%

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Figure 3.5 Average Number of Days to Third Next Acute Appointment (Primary Care) – By Service

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

 -

 1

 2

 3

 4

 5

 6

Average Number of Days

Month-Year

Air Force Army Navy NCR MD MHS Access Standard CA

43
August 29, 2014 3. Access to Care in the Military Health System

Facility Type: In general, the average number of days to third next acute appointment is higher 

in medical centers (2.6 days), compared to hospitals (1.5 days), and clinics (1.7 days) in FY 2014 

(see Figure 3.6).

Figure 3.6 Average Number of Days to Third Next Acute Appointment (Primary Care) – By Type of 

MTF

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Location: The average number of days to third next acute appointment is lower in overseas 

facilities compared to those in the United States in FY 2014: 0.8 days compared to 1.9 days. 

(See Appendix Figure 3.9-2.)

Routine Care 

Average Number of Days to Routine Appointment

This review does not include average days to routine appointments because the direct care 

component’s ability to schedule a routine appointment under different appointment categories 

makes data aggregation difficult. Due to the direct care component’s transformation to the 

PCMH model of primary care, 86 percent of routine appointments are classified as established 

(EST) appointments with the remaining 14 percent classified as routine (ROUT). As a result, 

routine access to care is evaluated through the Average Days to Third Next Routine Appointment 

in Primary Care measure, which includes both ROUT and EST appointment types. 

 -

 1

 2

 3

 4

 5

Average Number of Days

Month-Year

Clinic Hospital Medical Center MHS Access Standard CA

44
Military Health System Review – Final Report August 29, 2014

Average Number of Days to Third Next Routine Appointment in Primary Care

MHS Level: In FY 2014 to date, the average number of days to the third next routine 

appointment is 6.22 days, down 6 percent from 6.62 days in FY 2012 (see Figure 3.7). The CA 

access standard is 10 business days, which converts to 14 calendar days for comparison to the 

MHS access standard. The overall range of observations is 0.7 days to 12.5 days with 64 percent 

of MTFs performing better than the overall overage of 6.2 days. The median is 5.3 days and 

there were 3 minor outliers with average number of days to third next routine greater than 11.3 

days (72nd Medical Group, 45th Medical Group, 99th MDG Federal O’Callaghan) and no major 

outliers beyond 15.5 days. (See Appendix Table 3.10-3 and Appendix Figure 3.10-3.)

Figure 3.7 Average Number of Days to Third Next Routine Appointment (Primary Care) Overall

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Civilian Comparison: The average number of days to the third next routine appointment of 6.2 

days in direct care is better than Health System 2 and Health System 3, which averaged 9.0 days 

and 14.2 days, respectively. 

 -

 2

 4

 6

 8

 10

 12

 14

 16

Average Number of Days

Month-Year

DC Overall MHS Access Standard CA

45
August 29, 2014 3. Access to Care in the Military Health System

Service Level: The FY 2014 average number of days to the third next routine appointment is 5.6 

days for the Navy, 5.8 days for the Army, 6.9 days for the Air Force and 9.2 days for the NCR 

MD (see Figure 3.8). 

Figure 3.8 Average Number of Days to Third Next Routine Appointment (Primary Care) – By 

Service

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Facility Type: The average number of days to third next routine appointment is higher in 

medical centers (7.2 days), compared to hospitals (5.3 days) and clinics (5.7 days) in FY 2014 

(see Figure 3.9). 

Figure 3.9 Average Number of Days to Third Next Routine Appointment (Primary Care) –

By Type of MTF

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

 -

 2

 4

 6

 8

 10

 12

 14

 16

Average Number of Days 

Month-Year

Air Force Army Navy NCR MD MHS Access Standard CA

0

2

4

6

8

10

12

14

16

Average Number of Days

Month-Year

Clinic Hospital Medical Center MHS Access Standard CA

46
Military Health System Review – Final Report August 29, 2014

Location: The FY 2014 average days to the third next routine appointment at overseas facilities 

is 5.2 days compared to facilities in the United States at 6.3 days. (See Appendix Figure 3.9-3.) 

Specialty Care

Average Number of Days to Specialty Appointment

MHS: The FY 2014 average number of days to specialty appointment is 12.4 days, which 

outperforms the MHS access standard of 28 days and the CA specialty access standard of 15 

business days (21 calendar days) (see Figure 3.10). The overall range of observations is 6.5 days 

to 18.0 days with 67 percent of MTFs outperforming the overall overage of 12.4 days. The 

median is 11.6 days and there was one minor outlier with average number of days to specialty 

appointment greater than 15.7 days (52nd Medical Group) and no major outliers beyond 22.8 

days.29 (See Appendix Table 3.10-4 and Appendix Figure 3.10-4.)

Figure 3.10 Average Number of Days to Specialty Appointment – Overall:

MHS Access Standard ≤ 28 Day

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

29 Outliers for MHS-level data were calculated based on the interquartile range (IQR), which is the difference 

between the first and third quartiles of observed data. Minor and major outliers were defined as 1.5 times beyond 

IQR and 3 times beyond the IQR, respectively. See Appendix 3.10 for Outlier Analysis.

0

4

8

12

16

20

24

28

Average Number of Days

Month-Year

DC Overall MHS Access Standard CA

47

 
August 29, 2014 3. Access to Care in the Military Health System

Civilian Comparison: The average of 12.4 days to a specialty appointment in direct care is 

better than Health System 3, which reported an average days to specialty care appointments of 

22.3 days. The recent Merritt Hawkins survey of 15 major U.S. metropolitan areas reported a 

wait time to see a specialty provider of 18.5 days. 

Service-Level: The FY 2014 average number of days to a specialty appointment is 11.8 days for 

the Navy, 12.2 days for the Army, 13.1 days for the Air Force, and 14.7 days for NCR MD (see 

Figure 3.11). 

Figure 3.11 Average Number of Days to Specialty Appointment – By Service:

MHS Access Standard ≤ 28 Day

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014

0

4

8

12

16

20

24

28

Average Number of Days

Month-Year

Air Force Army Navy NCR MD MHS Access Standard CA

48
Military Health System Review – Final Report August 29, 2014

Facility Type: In FY 2014, the average number of days to specialty appointments in clinics, 

hospitals, and medical centers is 10.7 days, 12.8 days, and 13.6 days, respectively (see Figure 

3.12). All three facility types meet both the MHS access standard of 28 days and the CA 

specialty access standard of 15 business days (21 calendar days).

Figure 3.12 Average Number of Days to Specialty Appointment – By Facility Type: 

MHS Access Standard ≤ 28 Day

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Location: MTFs located overseas had only slightly better performance compared to facilities 

located in the United States. The FY 2014 average number of days to specialty appointment 

overseas is 11.5 days compared to 12.5 days in facilities located in the United States. Both 

groups performed better than the MHS access standard of 28 days and the CA specialty access 

standard of 15 business days or 21 days. (See appendix Figure 3.9-4.)

Average Days to Third Next Specialty Care Appointment

MHS Level: In FY 2014 to date, the average number of days to the third next specialty care 

appointment is 12.4 days (see Figure 3.13). The median is 11.5 with a range from 3.1 days to 

37.1 days. There is one minor outlier with average days to third next specialty appointments 

greater than 21.5 days but still within access standard (27th Special Operations Medical Group)

and one major outlier beyond 28.5 days (423 MDS-RAD Alconbury). (See Appendix Table 

3.10-5 and Appendix Figure 3.10-5.)

0

4

8

12

16

20

24

28

Average Number of Days

Month-Year

Clinic Hospital Medical Center MHS Access Standard CA

49
August 29, 2014 3. Access to Care in the Military Health System

Figure 3.13 Average Number of Days to Third Next Specialty Care Appointment – Overall

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Civilian Comparison: The average number of days to the third next specialty appointment of 

12.9 days in direct care is better than Health System 2, which averaged 16.7 days.

Service Level: The FY 2014 average number of days to the third next specialty care 

appointment averages 12.1 days for the Navy, 12.6 days, for the Army, 14.0 days for the Air 

Force, and 16.7 days for the NCR MD (see Figure 3.14). 

Figure 3.14 Average Number of Days to Third Next Specialty Care Appointment – By Service

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

0

5

10

15

20

25

30

Average Number of Days

Month-Year

DC Overall MHS Access Standard CA

0

5

10

15

20

25

30

Average Number of Days

Month-Year

Air Force Army Navy NCR MD MHS Access Standard CA

50
Military Health System Review – Final Report August 29, 2014

Facility Type: In general, the average number of days to third next specialty care appointment 

is higher in medical centers (14.2 days), compared to hospitals (12.1 days), and clinics (11.9 

days) in FY 2014 (see Figure 3.15). 

Figure 3.15 Average Number of Days to Third Next Specialty Care Appointment – By Facility Type

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

Location: The FY 2014 average number of days to third next specialty care appointment at 

overseas facilities is 11.5 days compared to facilities in the United States at 13.2 days. (See 

Appendix Figure 3.9-5 showing performance from August 2011 to May 2014.)

0

5

10

15

20

25

30

Average Number of Days

Month-Year

Clinic Hospital Medical Center MHS Access Standard CA

51
August 29, 2014 3. Access to Care in the Military Health System

TRICARE OnLine (TOL) Booking: Web-Enabled Appointments 

TOL is a web portal available to MTF enrollees for making appointments and for viewing 

personal health information 24 hours a day. Each month, more than 700,000 appointments, 

approximately 65 percent of available appointments, are web-enabled for TOL booking. This 

overall percentage of TOL web-enabled appointments has been stable over the past three years 

(see Figure 3.16). In the direct care component, nearly three percent of appointments are booked 

using TOL. This rate has not changed in three years. 

Figure 3.16 TRICARE On-Line (TOL) Web-Enabled Appointments: MHS Overall

2014 MHS Review Group

Source: TRICARE Operations Center (TOC), June 2014 

0%

20%

40%

60%

80%

100%

Percent Appts Web Enabled for TOL Booking

Month-Year

52
Military Health System Review – Final Report August 29, 2014

Secure Messaging (SM): Number of MTF Enrollees Registered 

The direct care component has enhanced access through SM, which broadens the relationship 

between the patient and their PCM. Patient satisfaction with SM is 97 percent and more than 86 

percent of SM satisfaction survey respondents agreed that using SM interaction allowed them to 

avoid an unnecessary trip to the clinic, emergency department, or urgent care facility. SM was 

fully implemented across primary care clinics in January 2014; specialty care implementation 

began in February 2014. As of June 2014, more than 1 million MTF enrollees were registered in 

SM, a growth of 53 percent in FY 2014 compared to FY 2013 (see Figure 3.17). 

Figure 3.17 Number of MTF Enrollees Registered in Secure Messaging – By Service, 

August 2010 – May 2014

2014 MHS Review Group

Source: Relay Health Reports, June 2014 

 -

 100,000

 200,000

 300,000

 400,000

 500,000

# Cumulative Enrollees Registered

Year-Month

Air Force Army Navy NCR MD

53
August 29, 2014 3. Access to Care in the Military Health System

Percent of Enrollees Registered in Secure Messaging Who Initiated Contact with PCM

Percentages-to-date by type of patient-initiated clinical messages are reported in Table 3.3. 

Table 3.3 Type of Patient-Initiated Clinical Message

Type of Patient-Initiated Message Percent of Total

Question for PCM 56%

Appointment Requests 17%

Rx Refill Requests 16%

Laboratory Result Requests 7%

Referrals 4%

2014 MHS Review Group

Source: Relay Health reports, June 2014 

The current direct care component policy is for patient-initiated messages to be answered within 

72 hours. The FY 2014 average response time is 35 hours. In FY 2014, 17 percent of users have 

initiated a clinical message to request an appointment (see Table 3.4). 

Table 3.4 Registered Users Initiating a Secure Message to PCM, FY14

MHS Group Percent of Total

MHS Overall 17%

Army 21%

Navy 15%

Air Force 14%

NCR MD 26%

2014 MHS Review Group

Source: Relay Health reports, June 2014 

54
Military Health System Review – Final Report August 29, 2014

Nurse Advice Line (NAL)

In March 2014, the MHS began the phased rollout of a 24-hour a day, 7-day a week NAL to 

enhance access to care for beneficiaries. All beneficiaries calling the NAL receive advice, health 

care finder assistance, and, if indicated, direct MTF primary care appointing. If an appointment 

cannot be obtained within the time recommended based on the triage end point and the clinical 

judgment of the NAL’s professional registered nurses, the patient will be advised to seek care in 

the network. A formal survey process will begin after full implementation; however, initial 

informal feedback from patients and staff has been overwhelmingly positive. Call volume is 

steadily increasing to an average of over 1,000 calls per day currently (see Figure 3.18).

On average to date, 12 percent of NAL callers were referred to private sector emergency 

departments, 25 percent to private sector urgent care facilities, 23 percent to the MTF (PCMH, 

urgent care [UC] or emergency department [ED]), and 40 percent to self-care (the patient did not 

need an appointment, just advice). If the MTF is unavailable, the NAL refers the patient to the 

private sector for urgent care. Notably, 1.3 percent of all calls to date have resulted in a decision 

to activate the emergency medical system (EMS) response. The most common reasons for 

activating EMS are chest pain, neurological problems, and breathing problems. 

Figure 3.18 Nurse Advice Line (NAL) Calls Triaged 28 March to 25 July 2014

2014 MHS Review Group

Source: NAL Live Web Repository, June 2014 

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

0

200

400

600

800

1,000

1,200

1,400

28-Mar 18-Apr 9-May 30-May 20-Jun 11-Jul

Number of Calls (Cumulative)

Number of Calls (per day)

Day - Month

Triaged Calls Per Day Cumulative

55
August 29, 2014 3. Access to Care in the Military Health System

Emergency Department and Urgent Care Center Utilization by MTF Enrollees in the 

Purchased Care Network

If primary care cannot be delivered in the MTF within MHS access standards, HA Policy 11-05 

states that patients are to be offered an appointment in the purchased care network. To assess 

access in the direct care component, the percent of care offered in both components for MTF 

enrollees was reviewed. In FY 2014, 79 percent of MTF enrollee primary care was delivered in 

the PCMH and 10 percent elsewhere in the MTF (direct care Emergency Department [ED] or 

Urgent Care [UC] resources). Approximately 11 percent of MTF enrollee primary care was 

delivered in the purchased care network: 4 percent was delivered in purchased care EDs and 7 

percent in purchased care UC. Since FY 2011, the percentage of MTF enrollee primary care 

delivered in the PCMH has increased from 73.5 percent to 78.6 percent while the percent of 

primary care delivered in the purchased care network (ED and UC) has decreased from 14 

percent to 11.1 percent (see Figure 3.19). 

Figure 3.19 MTF Enrollee Primary Care Workload, by Venue of Care, FY11 – FY14

2014 MHS Review Group

Source: Military Health System Mart (M2), June 2014 

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

FY11 FY12 FY13 FY14 (to May

14)

Percent of Enrollee Primary Care

Fiscal Year

PCMH Direct Care ED or UC PC ED PC UC

56
Military Health System Review – Final Report August 29, 2014

Patient Satisfaction Surveys

The Army, Navy and Air Force each measure patient satisfaction with access to care through 

Service-specific surveys. Additionally, the DoD evaluates patient satisfaction with access to care 

through the results of two surveys, TRICARE Outpatient Satisfaction Survey (TROSS) and 

Health Care Survey for DoD Beneficiaries (HCSDB).

Service-Specific Survey Results: Each Service has a patient satisfaction survey that asks about 

patient satisfaction with “getting care when needed.” Service surveys have averaged 85 percent 

satisfaction with this question over the last two years. Service-specific surveys average 636,000 

responses annually (25-percent response rate) (see Figures 3.20 and 3.21).

In the direct care component, analysis demonstrated a correlation between both lower average 

number of days to third next acute and third next routine appointments and higher PCM 

continuity. It also showed a lower average number of days to third next acute and third next 

routine each correlated with higher patient satisfaction. Finally, higher PCM continuity 

correlated with higher patient satisfaction. (See Appendix 3.11 for correlation analyses.)

Figure 3.20 Satisfaction with “Getting Care When Needed” (Service Surveys) Overall, FY12 Q1 –

FY14 Q1

2014 MHS Review Group

Source: Air Force Service Delivery Assessment (SDA); Army Provider Level Satisfaction Survey (APLSS); 

Patient Satisfaction Survey (PSS), June 2014 

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1

2012 2013 2014

Percent Satisfied

Fiscal Quarters

57
August 29, 2014 3. Access to Care in the Military Health System

Figure 3.21 Satisfaction with “Getting Care When Needed” (Service Surveys) – By Service*,

FY12 Q1 – FY14 Q1

2014 MHS Review Group 

Source: Air Force Service Delivery Assessment (SDA); Army Provider Level Satisfaction Survey (APLSS); Patient 

Satisfaction Survey (PSS), June 2014 

*Each Services’ patient satisfaction survey collects data using different modes. AHRQ reports that 

collection modes for patient satisfaction data produce significantly different response rates and 

satisfaction scores. 

TROSS: The TROSS is sent randomly to MHS beneficiaries following outpatient encounters 

with a MTF (including DHA facilities) or civilian provider. Respondents include TRICARE 

Prime enrollees and those not enrolled but eligible for TRICARE through other plan options, 

such as TRICARE Standard or Extra. Responses are defined as direct care if the patient received 

the outpatient care at an MTF and purchased care if the care was received from a civilian 

provider, regardless of the patient’s TRICARE plan. In this report, two TROSS questions are 

evaluated, one composite and one standalone question (Note: satisfaction with network access is 

analyzed in the purchased care section). TROSS metrics are compared to benchmarks 

established by AHRQ through the Consumer Assessment of Healthcare Providers and Systems 

(CAHPS). The MHS compares to the CAHPS 75th percentile for its TROSS access to care 

composite. There is no benchmark for the TROSS question asking whether the patient gets care 

when needed. (See Appendix 3.11 for specific questions asked on the TROSS and HCSDB 

including the CAHPS percentiles.)

TROSS Overall Access to Care: Over three quarters of FY 2013, the average satisfaction 

ratings for direct care matched the contemporaneous CAHPS benchmark of 60 percent (see 

Figures 3.22 and 3.23). Only the NCR MD demonstrated substantial variation, falling below the 

CAHPS benchmark. TROSS surveys average 64,700 responses annually with a response rate of 

19.2 percent for the direct care component.

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1

2012 2013 2014

Percent Satisfied

Fiscal Quarters

 Army Navy Air Force

58
Military Health System Review – Final Report August 29, 2014

Figure 3.22 TROSS – Satisfaction with Access to Care – Direct Care, FY11 Q1 – FY13 Q4

2014 MHS Review Group

Source: Department of Defense TRICARE Outpatient Satisfaction Survey (TROSS), June 2014

*Benchmark changes quarterly

Figure 3.23 TROSS – Satisfaction with Access to Care – By Service, FY11 Q1 – FY13 Q4

2014 MHS Review Group

Source: Department of Defense TRICARE Outpatient Satisfaction Survey (TROSS), June 2014 

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4

2011 2012 2013

Percent Satisfied

Fiscal Quarters

 Benchmark Direct Care

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4

2011 2012 2013

Percent Satisfied

Fiscal Quarters

 Air Force Army Navy NCR MD

59
August 29, 2014 3. Access to Care in the Military Health System

Seeing Provider When Needed. Figures 3.24 and 3.25 display patient satisfaction with “seeing 

provider when needed.” Across the direct care component, beneficiaries report higher 

satisfaction with “seeing provider when needed” than with their overall access. There is no 

CAHPS benchmark for this measure of satisfaction.

Figure 3.24 TROSS – Satisfaction with “Seeing Provider when Needed.” Direct Care, FY11 Q1 –

FY13 Q4 

2014 MHS Review Group

Source: Department of Defense TRICARE Outpatient Satisfaction Survey (TROSS), June 2014 

Figure 3.25 TROSS – Satisfaction with “Seeing Provider when Needed,” by Service, Direct Care, 

FY11 Q1 – FY13 Q4 

2014 MHS Review Group

Source: Department of Defense TRICARE Outpatient Satisfaction Survey (TROSS), June 2014

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4

2011 2012 2013

Percent Satisfied

Fiscal Quarters

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4

2011 2012 2013

Percent Satisfied

Fiscal Quarter

Air Force Army Navy NCRMD

60
Military Health System Review – Final Report August 29, 2014

TROSS Patient satisfaction with “seeing provider when needed” is higher than overall 

satisfaction with access. This indicates that satisfaction with supplemental ATC measures 

included in the overall access composite, such as time spent in the waiting room and 

communication with a provider, contribute substantially to the overall access to care score. The 

degree to which these factors contribute to a patient’s perception of access should be further 

analyzed.

HCSDB: HCSDB is sent randomly to all MHS-eligible users and non-users, independent of 

whether they had a recent encounter. Respondents include those enrolled to TRICARE Prime 

(MTF and network enrollees) and non-enrolled beneficiaries who may receive care in MTFs or 

through the purchased care system. For this report, only the HCSDB results for Prime enrollees 

are presented (Note: Prime beneficiaries enrolled to the network are presented in the purchased 

care section). Beneficiary responses to two composite questions that address the beneficiary’s 

ability to “get care quickly” and “get needed care” are evaluated. The HCSDB response rate is 

18 percent. (See Appendix 3.12 for questions and CAHPS benchmarks.) 

HCSDB Getting Care Quickly. Satisfaction with “getting care quickly” has remained 

relatively constant over time. Over the past four years MTF-enrollee satisfaction ranges from 71 

to 74 percent and is below the CAHPS benchmark of 86 percent. Figures 3.26 and 3.27 display 

Service-level satisfaction with “getting care quickly”. Three of the Services remained relatively 

constant over time, although the NCR MD demonstrates variation in satisfaction and has lower 

satisfaction with “getting care quickly” than the other Services. There were no HCSDB data 

available in FY 2013 Q4. No information was available on the number of HCSDB respondents 

or response rate.

Figure 3.26 HCSDB – Satisfaction with “Getting Care Quickly”, FY10 Q1 – FY14 Q2

2014 MHS Review Group

Source: Health Care Survey of Department of Defense Beneficiaries (HCSDB), June 2014

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2

2010 2011 2012 2013 2014

Percent Satisfied

Fiscal Quarters

 Benchmark USA MHS Direct Care

61
August 29, 2014 3. Access to Care in the Military Health System

Figure 3.27 HCSDB – Satisfaction with “Getting Care Quickly” – by Service, FY10 Q1 – FY14 Q2

2014 MHS Review Group

Source: Health Care Survey of Department of Defense Beneficiaries (HCSDB), June 2014 

HCSDB Getting Care When Needed. Patient satisfaction with “getting care when needed” 

does not differ greatly from satisfaction with “getting care quickly.” Over the past four years, 

MTF-enrolled patient satisfaction ranges from 71 to 76 percent and has remained below the 

CAHPS benchmark of 85 percent in each quarter. Figures 3.28 and 3.29 display Service-level 

satisfaction with ‘getting care quickly”. On average, the Services have remained at or above 70 

percent satisfaction for the past four years while the NCR MD remained below 70 percent until 

the most recent survey quarter. No information was available on the number of HCSDB 

respondents or response rate.

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2

2010 2011 2012 2013 2014

Percent Satisfied

Fiscal Quarters

 Air Force Army Navy NCR MD

62
Military Health System Review – Final Report August 29, 2014

Figure 3.28 HCSDB – Satisfaction with “Getting Care When Needed”, FY10 Q1 – FY14 Q2

2014 MHS Review Group

Source: Health Care Survey of Department of Defense Beneficiaries (HCSDB), June 2014 

Figure 3.29 HCSDB – Satisfaction with “Getting Care when Needed” – by Service, FY10 Q1 –

FY14 Q2

2014 MHS Review Group

Source: Health Care Survey of Department of Defense Beneficiaries (HCSDB), June 2014 

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2

2010 2011 2012 2013 2014

Percent Satisfied

Fiscal Quarters

 Benchmark USA MHS Direct Care

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2

2010 2011 2012 2013 2014

Percent Satisfied

Fiscal Quarter

 Benchmark Air Force Army Navy NCR MD

63
August 29, 2014 3. Access to Care in the Military Health System

Purchased Care

TRICARE Prime beneficiaries are entitled to seek medical care in the purchased care sector if 

that care is not available at the MTF. The current TRICARE contracts do not require TRICARE 

regional contractors to provide the same level of data on access to care as is available in the 

direct care component. The TRICARE regional contractors do not collect primary care access 

data; however, they do report the percent of specialty appointments meeting the MHS access 

standard. In addition, there is variation in reporting among the TRICARE regional contractors.

Each of the three regional contractors in the United States is required to develop a network of 

contract providers to serve Prime beneficiaries living within 40 miles of an MTF (Prime Service 

Areas [PSA]). This same network of providers also serves as the TRICARE Extra network for 

beneficiaries using TRICARE Standard. Although regional contractors are not required to 

contract with providers outside of PSAs, they are encouraged to do so to further expand the 

TRICARE Extra network. To evaluate whether patients receive a high degree of access to 

network providers in the United States, several measures are analyzed, including network 

adequacy (number of network providers and drive time to a provider), 28-day Access Standards 

Reports provided by the contractors, and patient satisfaction. 

Overseas network care is managed by the TOP contractor, International SOS. Data indicating 

whether 28-day access standards are met are not available; however, the robustness of a network 

can be assessed based on the number of contracted providers and the percentage of claims paid to 

network providers. Survey data are also collected to determine whether patients are satisfied 

with network care overseas. Like in the United States, if a specialist is not available, the 

contractor is contractually responsible for locating a non-network provider. 

Network Adequacy (United States)

In the United States, the number of contract network providers within a PSA is based on the 

TRICARE-eligible beneficiary population within that PSA. Figures 3.30 and 3.31 show the 

number of providers per enrolled beneficiary within each region’s PSAs. Most of the increase in 

the number of providers per enrolled beneficiaries is largely attributed to the change to the new 

contractor in the West region, which has a larger network of providers. It should be noted that 

these figures represent network providers. In cases where a contracted specialist is not available, 

the contractor has contractual responsibility to locate a non-network provider for Prime-enrolled 

beneficiaries. Since FY 2011, more than 97 percent of referrals were to a network provider or 

the MTF in each of the three regions, with only 3 percent to non-network providers.

64
Military Health System Review – Final Report August 29, 2014

Figure 3.30 Primary Care Providers per 1,000 Enrolled Beneficiaries (Restricted to beneficiaries 

living in Prime Service Areas), FY11 Q1 – FY13 Q3

2014 MHS Review Group

Source: Regional Managed Care Support Contractors Data System, June 2014

Figure 3.31 Specialty Care Providers per 1,000 Enrolled Beneficiaries (Restricted to beneficiaries 

living in Prime Service Areas), FY11 Q1 – FY13 Q3

2014 MHS Review Group

Source: Regional Managed Care Support Contractors Data System, June 2014

0

5

10

15

20

25

Primary Care Providers per 1,000 enrollees

Fiscal Quarter

North South West

0

20

40

60

80

100

120

Specialty Care providers per 1,000 enrollees

Fiscal Quarter

North South West

65
August 29, 2014 3. Access to Care in the Military Health System

The percentage of patients driving greater than 60 minutes for a specialist appointment is another 

measure to evaluate network adequacy. Contractors are required to refer patients to non-network 

providers before requiring the patient to drive greater than 60 minutes. If a specialist is not 

available within 100 miles of the patient’s PCM, the Prime Travel Benefit provides 

reimbursement for travel expenses. Figure 3.32 indicates that in 81 of the 102 PSAs, patients are 

able to see a specialist within 60 minutes of their home over 90 percent of the time. In six PSAs, 

patients must drive more than 60 minutes over 25 percent of the time.

Figure 3.32 Percent of Patients in a PSA Required to Drive More than 60 Minutes to see a 

Specialist

2014 MHS Review Group

Source: Regional Managed Care Support Contractors Data System, June 2014

28-Day Access to Network Care (United States)

Network compliance with specialty care access standards is measured by the percentage of 

specialty referrals with an appointment within 28 days. This measure includes all referrals 

originating within an MTF and sent to a network provider, as well as those referred from within 

the network. This metric does not take into account the administrative time to process the 

referral and is based on when the patient made the appointment; these two factors may contribute 

to the appointment being outside the 28-day standard. The 28-day access standard is tracked 

within the 102 PSAs. Figure 3.33 plots the percentage of referrals meeting the 28-day access 

standard for each PSA. In FY 2013, an average of 68 percent of specialty appointments met the 

MHS access standard and there is an upward trend in the number of specialty appointments 

meeting the 28-day standard. There is variation across PSAs with averages ranging from 53 

percent to 84 percent.

0

20

40

60

80

100

0-10% 10-25% 25-50% >50%

Number of PSAs

Percent of patients in the PSA required to drive more 

than 60 minutes to see a specialist

66
Military Health System Review – Final Report August 29, 2014

Figure 3.33 Percentage of Network Referrals Meeting 28 Day Access Standard, by PSA 

(Each point represents one PSA), FY11 Q1 – FY14 Q4

2014 MHS Review Group

Source: Regional Managed Care Support Contractors Data System, June 2014

Patient Satisfaction with Network Access to Care (United States)

Satisfaction with access to care in the network is measured by TROSS and HCSDB. The results 

in Figure 3.34 and Table 3.5 demonstrate that patient satisfaction with access to care in the 

network is consistently close to CAHPS benchmarks. Note that HCSDB results are available 

through Q2 of FY 2014, but TROSS results are only available through FY 2013. The break 

present in the HCSDB results is because the survey was not fielded in Q4 of FY 2013.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percent of appointments meeting standard

0 2 4 6 8 10 12 14

Fiscal Quarter

FY11-Q1FY11-Q2 

FY11-Q3 FY11-Q4 FY12-Q1 

FY12-Q2 FY12-Q3 FY12-Q4 FY13-Q1 FY13-Q2 

FY13-Q3 FY13-Q4 

67
August 29, 2014 3. Access to Care in the Military Health System

Figure 3.34 U.S. Network Satisfaction: Four Satisfaction Measures, FY10 Q1 – FY14 Q1

2014 MHS Review Group 

Source: Department of Defense TRICARE Outpatient Satisfaction Survey (TROSS), June 2014

Table 3.5 CAHPS Benchmarks

Benchmark Percent

TROSS-Access to Care Composite (FY 2013) 60%

HCSDB-Get Care Quickly (FY 2014) 86%

HCSDB-Get Care When Needed (FY 2014) 85%

2014 MHS Review Group

Source: Consumer Assessment of Healthcare Providers and Systems, June 2014 

Access to Network Care (Overseas)

Network adequacy overseas is measured by the number of providers and the percent of referrals 

seen by a network provider. The total number of network providers has remained stable since 

2012 at approximately 9,300, and more than 83 percent of the claims processed were from TOP 

network providers. Any TOP-enrolled beneficiary who cannot be appointed to a network 

provider is referred to a non-network provider.

Overseas network satisfaction is also evaluated through patient survey data. The two questions 

are:

1) If you were referred to a specialist in the last 60 days, how satisfied were you with 

International SOS’ coordination of the referral?

40%

50%

60%

70%

80%

90%

100%

Percent Satisfied

Fiscal Quarter

TROSS- Saw Provider When Needed TROSS- Access to Care Composite

HCSDB- Get Care Quickly HCSDB- Get Care When Needed

68
Military Health System Review – Final Report August 29, 2014

2) If you received services from a civilian network provider, how satisfied were you with 

the service you received? 

Figures 3.35 and 3.36 display quarterly results for these two questions. Responses are measured 

on a 1-6 scale, with 6 being “completely satisfied.” The mean satisfaction score for civilian 

network providers and specialists is close to 5, increasing slightly over 3 years. Since this is a 

unique military contract, the data can only be compared with previous satisfaction surveys. 

Figure 3.35 Satisfaction with International SOS Coordination of Specialty Care, FY11 Q2 – FY13 Q4

2014 MHS Review Group

Source: TRICARE Overseas Program Contractor Data Systems, June 2014

3

4

5

6

Level of Satisfaction

Fiscal Quarter

Eurasia/Africa Pacific Latin America-Canada

69
August 29, 2014 3. Access to Care in the Military Health System

Figure 3.36 Satisfaction with Civilian Network Care, FY11 Q2 – FY13 Q4

2014 MHS Review Group

Source: TRICARE Overseas Program Contractor Data Systems, June 2014

Site Visit Information

Introduction

The site visits were conducted to validate if the access to care data correlated with execution of 

policy, performance measures, and patient experiences. The site visits demonstrated that MTF 

executive leadership, as well as clinic leadership and staff, are aware of their patients’ 

experiences and perceptions of access at the majority of sites. It was also apparent that there is a 

patient-centered culture driven by the leadership of every facility, as witnessed during the 

executive leadership sessions. 

The patient-centered culture was repeatedly observed when clinics found a way to get a patient 

the care they needed despite there being limited or no appointments available. Rather than ask 

the patient to call back at a later time or date, scheduling clerks, nurses, and providers would 

collaborate to identify options for the patient to obtain care. As an example of patientcenteredness, one MTF provides patients with historical average wait times for network care to 

assist the patient in making an informed decision about where they would like to be seen. It is 

standard practice across all seven MTFs to schedule any patient follow-up visit at the clinic 

checkout desk before the patient departs. If this was not possible (e.g., the requested follow-up 

appointment date was beyond the published schedule), a reminder list was employed by several 

clinics. Though these lists were not used by all MTFs, those that were used were managed 

appropriately, allowing the MTF to effectively facilitate future appointments.

 

3

4

5

6

Level of Satisfaction

Fiscal Quarter

Eurasia/Africa Pacific Latin America-Canada

70
Military Health System Review – Final Report August 29, 2014

Most sites had a centralized access management group that monitored all clinics and provided 

regular access analysis and training down to the clinic leadership level. Staffing of these 

functions varied among sites. Although leadership was keenly aware of the information the 

access to care management group provided, most of the frontline staff was unaware of clinic 

performance.

Regional and Headquarters Leadership Survey

The Regional Headquarters leadership completed a survey to capture its perspectives on 

subordinate facility performance and processes. The survey included eight questions on access, 

five scored on an objective Likert scale and three with a narrative. The Likert scale responses 

from worst to best: Not at all, partially, neutral, effective, and very effective. On the five 

objective questions, the headquarters indicated an 83-percent level of confidence that subordinate 

MTFs performance relative to access is “effective” or “very effective.” On the narrative 

questions, the regional level reported focused efforts on regular performance reviews with 

initiatives to improve access. 

On-site: Leadership, Staff, Patients

MTF leadership demonstrated several initiatives focused on improving access to care, including

informing executive and other MTF staff members on performance to drive improvement. These 

initiatives included presentations to the executive leadership by clinic managers, provider 

scorecards based on performance metrics, and updates at weekly clinical business forums. Other 

initiatives focused on changing daily operations to be more patient-centered, including

modification of operating hours and provider duty hours as well as employing providers 

dedicated to seeing walk-ins. 

MTF leadership and staff across the majority of facilities were aware of specific access issues, 

which appeared to be aligned with their patients’ perceptions of access. Most of the MTFs relied 

on the purchased care network to meet patients’ same day primary care needs when 

appointments were no longer available. While all sites had processes for referring to either an 

UC center or neighboring MTF, the majority of patients expressed that they prefer to be seen in 

the MTF and are more satisfied with the care they receive when seen by their PCM. Same day 

access is also limited due to staff members converting same day appointments to accommodate 

follow-up needs. In a few clinics, same day appointments are only opened first thing in the 

morning, so patients are told to call back to get a same day appointment. While staff usually 

followed the UC referral processes when unable to offer an appointment during the initial 

contact, staff and patients acknowledged that patients are asked to call back on occasion. 

Additionally, the move to a simplified appointment system in primary care has caused confusion 

in the use of specific appointment types related to appointment time frames (routine versus 

follow-up care). This confusion was not apparent in specialty clinics; however it was noted that 

there are no standards for the time to procedure. 

The greatest challenge faced by the smaller facilities visited is staffing shortages, some of which 

are brought on by the annual permanent change of station season, but also by other unavoidable 

events, including resignations, furloughs, retirements, illness/injury, and deployments. Same day 

71
August 29, 2014 3. Access to Care in the Military Health System

appointment availability is most heavily affected by these shortages. Additionally, the civilian 

employee hiring process is lengthy and cumbersome, which is a key factor in access degradation 

in some of these facilities. 

Patients at a few sites stated that access to specialty care within the network or other direct care 

facilities in the area was inconvenient. While the complaint was related to the driving distance 

involved either in highly congested areas or very remote areas, it was still within the 60-minute 

drive time standard. One active duty member from an operational platform expressed frustration 

at navigating the system to see a specialist. 

TOL and SM are demand management tools that can be used to reduce face-to-face 

appointments and access demand, in addition to increasing patient convenience. There was 

variance in prioritizing web-enabled appointing and education of TOL. On the other hand, SM is 

becoming increasingly useful to MTFs and the majority of sites are aggressively marketing its 

functionality for patients. MTFs have increased enrollment in SM by 38 percent in the last year. 

All MTFs would benefit from increased marketing and utilization of both the TOL appointing 

function and the use of SM for enhanced patient access to primary care; increased TOL 

appointing allows patients to book 24 hours a day, seven days a week, rather than just when the 

appointment desk is open during duty hours and allows patients to communicate and have virtual 

visits with their PCM in lieu of face-to-face visits.

Site Visit Survey Results

In reviewing the data from the site visit surveys, it is evident that regional headquarters and MTF 

leadership are aligned in their understanding of access within the MTF (Figure 3.37). At most 

MTFs, there was equal knowledge of access practices and performance between MTF mid-level 

managers and the clinic staff interviewed. At all but two sites, the perspectives of the subject 

matter experts (SMEs) and staff are in line with the patients’ perspectives. Of note, at one of 

these two sites, patients and staff had a more positive view of access than did the SMEs.

72
Military Health System Review – Final Report August 29, 2014

Figure 3.37 Access: Perceptions Among Regional Headquarters, MTF Leaders, Subject Matter 

Experts (SMEs), Staff Members and Patients During Seven MHS Site Visits, 2014

2014 MHS Review Group

Source: MHS Site Visit Survey, June - July 2014

Staff Town Halls

Staff town halls were conducted to assess staff knowledge and experience with access policies 

and procedures. The theme across all sites indicated there are various levels of success and 

challenges in meeting access standards. Staffing shortages (seasonal and persisting) and 

variations in schedule management practices among clinics were most often cited as challenges 

to meeting demand. Several comments indicated that schedules fill up very quickly after being 

released. Staff in primary care and specialty care often had different perspectives and 

challenges. Staff demonstrated good understanding of access issues within their areas and is

doing everything within their control to take care of the beneficiary.

Beneficiary Town Halls

Beneficiary town halls included a mix of beneficiary categories and were conducted to assess 

experience with access to care at the MTF and in the purchased care component. An identified 

theme at the site visits was that patients expressed difficulty in obtaining appointments. Of the 

100 comments about access, 29 percent of responses were positive and 71 percent were negative. 

Identified problem areas included difficulty in getting an appointment when wanted, being told 

73
August 29, 2014 3. Access to Care in the Military Health System

to call back for an appointment, or difficulty in obtaining a specialty appointment. A variety of 

primary and specialty care clinic types were identified at town halls as presenting access 

challenges; no single clinic type was consistently identified. Some beneficiaries expressed 

challenges when they called the MTF appointment line and were more successful in obtaining an 

appointment when they went to the clinic in person. In addition, beneficiaries at several sites 

reported they were referred to an ED or UC center to obtain care when acute appointments were 

not available at the MTF. Beneficiaries reported SM is a welcome addition and a useful option 

for contacting their provider. They also expressed a wish that more providers would use SM. 

Overall, the participating beneficiaries expressed a range of experiences that identified areas of 

MTF performance that should be sustained and others that should be improved. 

Access to Care: Overall Findings and Recommendations

Based on the analysis of MHS access data, the MHS provides ready access to medical care as 

defined by access standards in policies and guidance of HA, the military medical departments, 

and in TRICARE contract specifications. In accordance with DoD’s access standards, MTFs 

have made significant progress in the last three years to increase MTF access to care capacity 

within the direct care component. The majority of patients in the direct care component receive 

medical care within MHS access standards. In addition to face-to-face encounters, the direct 

care component has multiple modalities for accessing care and assistance, including Secure 

Messaging, TRICARE On-Line, and a purchased care safety net. Satisfaction surveys 

demonstrate that the majority of patients reported being satisfied with access to care. There is 

variance between satisfaction scores in the direct care and purchased care components, 

depending on which survey tool is used. In addition, the town hall respondents reported 

instances of access challenges, which present opportunities for further exploration and 

improvement. Access policy has achieved significant standardization in primary care over the 

past four years. Vertical alignment in access policy is noted, and site visits revealed a strong 

patient-centered culture across all levels of staff.

Specific findings are provided below:

1. The MHS provides ready access to medical care as defined by access standards in the 

policies and guidance of HA, the military medical departments, and in TRICARE 

contract specifications.

2. The review looked at specialty care as a whole, not individual product lines. Variance in 

specialty care business practices was noted on site visits.

3. There is variation in business process standardization, as evidenced in town hall meetings 

where some patients reported difficulty in getting an appointment or were asked to call 

back for an appointment.

4. 32 C.F.R. § 199.17 requires a level of detail not available for the purchased care 

component under current TRICARE contracts. While data are limited, the surrogate 

access measure is patient satisfaction. There are no access data available for non-Prime 

enrolled beneficiaries (TRICARE Standard / Extra). In the review, it was noted that the 

GAO and DoD IG reports focused on access for non-enrolled beneficiaries in the 

74
Military Health System Review – Final Report August 29, 2014

purchased care component; however, none focused on access for TRICARE Prime 

enrollees. 

5. There are multiple patient satisfaction survey tools (Service-specific surveys, TROSS, 

HCSDB) used across the MHS with varying response rates and results. 

6. There is variation in reporting of purchased care access data from each of the TROs to the 

Services.

7. There is no standardized MHS measure for evaluating office waiting times, as required 

by 32 C.F.R. § 199.17. 

8. There is variation in the promotion of SM and TOL as methods for enhanced access to 

care in the direct care component.

9. Each Service has developed its own training courses for access, clinic, and group practice 

management. The Services’ customer service training is not standardized.

Recommendations to Improve Access to Care

1. MHS governance should increase the focus on the standardization of specialty care 

in the direct care component through the following: a) create the Tri-Service 

Specialty Care Advisory Board, b) fund requirements to standardize specialty 

product lines, c) establish business rules for access, and d) define performance 

review metrics for specialty care product lines. 

2. MHS governance should standardize MHS direct care component access to care 

business practices by replacing the MHS Guide to Access Success with a MHS 

policy memorandum and subsequent DoD Instruction. 

3. MHS governance should commission an external study to evaluate purchased care 

access for TRICARE Prime enrollees as it relates to 32 C.F.R. § 199.17. This study 

should include a review of all available data and recommend metrics for 

incorporation into current and future TRICARE contracts. 

4. MHS governance should continue implementation of the Joint Service survey tool, 

refining access satisfaction questions to include satisfaction with office wait times.

5. MHS governance should standardize reporting on access from the TRICARE 

Regional Offices to the Services. 

6. MHS governance should promote Secure Messaging and TRICARE On-Line 

through direct care component standardized business processes and a strategic 

marketing approach. 

7. MHS governance should standardize both access to care and customer service 

training across the direct care component.

75
August 29, 2014 3. Access to Care in the Military Health System

This page left intentionally blank.

76
Military Health System Review – Final Report August 29, 2014

4. QUALITY OF CARE IN THE MILITARY HEALTH SYSTEM

Introduction

The Military Health System (MHS) has embraced the Institute of Medicine (IOM) definition of 

quality as “the degree to which health care services for individuals and populations increase the 

likelihood of desired health outcomes and are consistent with current professional knowledge.”30 

The IOM has set forth six aims for improving the delivery of health care: “safe, effective, 

patient-centered, timely, efficient and equitable.”31 All Department of Defense (DoD) policies 

and implementation guidance regarding health care quality are based on these IOM constructs as 

stated in Health Affairs (HA) Policy 02-01632 and further elaborated in DoD Instruction and 

Manual 6025.13.33 34

HA Policy 02-016 frames the assessment and review of the MHS quality of health care. First, is 

the foundation for the delivery of high-quality care in place and robust? Second, what are the 

measurable process and outcomes of care performance of DoD’s health care system? Third, how 

is the health care delivery system and the quality of health care provided viewed by DoD 

beneficiaries, military leadership, and Congress? 

It is the goal of this review to determine if the MHS meets or exceeds benchmarks for health care 

quality as defined by the Office of the Secretary of Defense (OSD), Service policies and 

guidance, and TRICARE contracts.

Quality of Care Governance 

Issues related to the quality of clinical care are brought to the MHS governance through the 

Clinical Quality Forum (CQF), a collaborative forum with representation from the direct care 

and purchased care components. The CQF has the responsibility for assessing clinical quality

across the MHS, based on relevant clinical indicators for health care system performance, 

including beneficiary and stakeholder perceptions of care, and activities focusing on quality 

improvement, patient safety, and risk management events. Additionally, the CQF develops, or 

may endorse, recommendations for clinical quality improvement for approval by Medical 

Operations Group (MOG), Medical Deputies Action Group (MDAG) or Senior Medical 

30 Institute of Medicine. (1990). Medicare: A Strategy for Quality Assurance. Vol. 1. Washington, D.C: National 

Academy Press. 

31 Institute of Medicine. (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. 

Washington, D.C: National Academy Press. 

32 Health Affairs. HA Policy 02-016: Military Health System Definitions of Quality in Health Care. May 2002. 

33 Department of Defense. DoD Manual 6025.13: Medical Quality Assurance (MQA) and Clinical Quality 

Management in the Military Health System (MHS), October 2013. 34 Department of Defense. DoD Instruction 6025.13: Medical Quality Assurance (MQA) and Clinical Quality 

Management in the Military Health System (MHS), February 2011.

77

 
August 29, 2014 4. Quality of Care in the Military Health System

Management Advisory Committee (SMMAC) (see Section 2 of this report regarding MHS

governance). 

Following the establishment of the Defense Health Agency (DHA), the CQF began reporting its 

recommendations for clinical quality improvements to the MOG, which in turn engages in a 

process to evaluate recommendations and convey those deemed appropriate to senior military 

leadership.

Each of the Services implements quality improvement efforts at its respective MTFs in line with 

its organizational processes through Joint Operational Commands, Regional commands, or in the 

case of the Air Force, Field Operating Agencies.

Policy Review

Subject matter experts (SMEs) from the Departments of the Army, Navy, Air Force, and DHA 

conducted a review of documents addressing the quality of care in the MHS, to include: 1) 

legislation that sets authority for the MHS and DoD on matters of quality, 2) DoD issuances 

addressing quality of care within DoD or the Services, 3) DoD publications guiding the delivery 

of services, including memoranda, operational or implementation guidance, 4) studies examining 

quality of care, 5) studies or reviews pertaining to quality of care in the MHS performed by the 

Government Accountability Office (GAO) after 2003, and 6) studies and reviews performed by 

the Department of Defense Inspector General (DoD IG) after 2003. The review focused on 

whether policies were within the scope of quality of care and identified the policy intent and 

agencies responsible for implementation (see Table 4.1). These policies and practices were 

contrasted among Services and differences identified between overarching DoD requirements 

and Service policies. (For a complete list of policies reviewed for this report, see Appendix 4.1.) 

Table 4.1 Numbers of Policies, Instructions, Resources and Manuals Reviewed

Level of Authority Number Reviewed

Legislation 8

Department of Defense 18

Army 28

Navy 14

Air Force 20

National Capital Region 1

 Source: 2014 MHS Review Group, July 2014

78
Military Health System Review – Final Report August 29, 2014

DoD Policy Guidance

DoD policy is guided by statutory and regulatory requirements: 

• 10 United States Code (U.S.C.) 55 § 1102 defines medical quality assurance records, peer 

review, and the type and extent information is protected from disclosure outside DoD.

35

• 32 Code of Federal Regulations (C.F.R) 199.15, “Quality and Utilization Review Peer 

Review Organization Program” establishes rules and procedures to review the quality, 

completeness, and adequacy of purchased care provided, as well as its necessity, 

appropriateness and reasonableness. Furthermore, it defines the requirements for external

peer review in the purchased care component. 

DoD Instruction 6025.13 and DoD Manual 6025.13, “Medical Quality Assurance (MQA) and 

Clinical Quality Management in the MHS,” provide the Services with guidance on clinical 

quality. DoDM 6025.13 establishes medical quality assurance programs, implements policy, 

assigns responsibilities, and provides procedures for managing DOD Medical Quality Assurance 

(MQA) and clinical quality management. This manual requires the implementation of a 

performance measurement system for clinical quality in every MTF as a dedicated program to 

confirm quality-of-care outcomes and identify opportunities for improvement. 

DoDI 5010.43, “Implementation and Management of the DoD-Wide Continuous Process 

Improvement/Lean Six Sigma (CPI/LSS) Program,” establishes policy, assigns responsibilities 

and provides guidance for implementation of CPI/LSS programs. It requires that all DoD 

components implement CPI/LSS as an essential tool for improving their operating effectiveness, 

apply its methodologies and practices to ensure cost-effective management, and implement 

enhanced processes and technologies. Furthermore, this instruction requires that demonstrated 

performance improvements be documented in a transparent fashion for managerial review, 

assessment, and knowledge sharing. 

HA Policy 10-008, “Policy Memorandum for Military Health System Health Care Quality 

Assurance Data Transparency,” requires the MHS to ensure that quality assurance information 

is clear, transparent, and readily available to MHS providers, as well as to its beneficiaries. 

Army Policy Guidance

Army Regulation (AR) 40-68 implements DoDI 6025.13 guidance on care quality, DoDM 

6025.13, DoDD 6000.14, and other DoD guidance. This consolidated regulation prescribes 

policies, procedures and responsibilities for the U.S. Army Medical Department (AMEDD) 

Clinical Quality Management Program (CQMP). The CQMP continuously and objectively 

35 The peer review process refers to the assessment of the quality of medical care carried out by a health care 

professional, including any such assessment of professional performance, any patient safety program root cause 

analysis or report, or any similar activity.

79

 
August 29, 2014 4. Quality of Care in the Military Health System

assesses individual and institutional performance, aiming to improve the health care provided to 

eligible DoD beneficiaries. 

AR 40-68 includes DoD and statutory policies addressing medical services quality management 

requirements and organizational performance improvements. It refers to the Joint Commission 

(TJC) requirement for an organized, self-governing medical staff to provide direction and 

oversight of the quality of care, treatment, and services delivered by privileged providers. 

Separately, AR 5-1, “Management of Army Business Operations,” further establishes policy and 

responsibilities for management and measurement of business operations, including Medical 

Command (MEDCOM) quality-of-care efforts, and continuous improvement efforts.

The Army Medicine 2020 Campaign plan establishes the framework through which AMEDD 

will achieve a responsive and reliable health service and move toward an integrated and 

standardized process across the organization with clear performance metrics. The Army 

Information Paper “Quality and Patient Safety Program Army Medical Department (AMEDD),”

describes the process to oversee and enforce AMEDD Quality and Patient Safety policies. The 

Clinical Performance Assurance Directorate (CPAD) monitors that process and reports directly 

to the Deputy Commanding General of Operations. The CPAD provides education, training, 

standardization, and leadership visibility to AMEDD’s Quality, Patient Safety and Risk 

Management programs. The CPAD provides professional education to all levels of leadership 

through courses at the AMEDD Center and School. CPAD provides monthly educational Video 

Teleconferences (VTC) with all facility quality leads and provides newsletters, milBook, and 

SharePoint sites to support mentoring at the MTF level. Additionally, CPAD monitors measures 

of quality within MTFs, Services, and the MHS, comparing them to civilian benchmarks. CPAD 

further submits near real-time actionable data to the MTFs and devises specific interventions 

based on data collected. Lessons learned are collected through MHS-level committees and 

pushed to the Regional leads for dissemination to the field. The CPAD combined with the 

MEDCOM command inspection program and other self-inspection systems ensure the execution 

and implementation of policies. CPAD is the MEDCOM lead to develop initiatives, policies, 

and standards, in close collaboration with the MHS, DHA, Air Force, and Navy. 

Navy Policy Guidance

BUMED Instruction 6010.13 provides guidance for Navy’s Quality Assurance (QA) program. It 

applies to all health care personnel providing services in naval MTFs, outlines basic component 

activities and functions, and requires all medical personnel to participate in ongoing monitoring 

and evaluation to assess quality of care provided. Furthermore, BUMED 6010.13 requires all 

MTFs to implement a QA program and maintain TJC accreditation. This QA program is 

intended to identify patient care improvement opportunities, identify and decrease risk to patients 

and staff, and provide justification for resource needs. It also monitors and analyzes QA data to 

identify patterns or processes requiring additional scrutiny. 

A separate instruction, BUMED 6000.2E, further establishes policy, publishes procedures, and 

assigns responsibility for the accreditation of Navy MTFs. It requires that all fixed MTFs and 

80
Military Health System Review – Final Report August 29, 2014

freestanding ambulatory clinics achieve and maintain TJC accreditation. Possible accreditation 

programs include Hospital, Ambulatory Care, Behavioral Health Care, and Home Care. 

The Navy Surgeon General is responsible for policy and initiatives to support higher authority 

mandates. Within the Bureau of Medicine and Surgery (BUMED) Clinical Operations 

Directorate (BUMED-M3) TJC survey reports are analyzed and actionable recommendations are 

made. Between triennial TJC surveys, a trained team of Navy TJC fellows completes monitoring 

of compliance with standards and quality goals. Interpretation, feedback, consultation on 

findings, and any identified performance issues are submitted to senior leaders. Validation of 

standards compliance is ongoing.

Air Force Policy Guidance

The Air Force Surgeon General (AF/SG)36 assists the Secretary of the Air Force in development 

of policies, plans, and programs, establishing requirements, and providing resources to the Air 

Force Medical Service (AFMS). The AF/SG prepares policies and issues official guidance and 

procedures to ensure implementation of those policies. The AF/SG manages Quality and Safety 

programs through the Air Force Medical Operations Agency (AFMOA) and the Directorate of 

Healthcare Operations (HAF/SG3). Elements of quality and patient safety are integrated 

throughout AFMS governance and are incorporated into the AFMS Strategic Plan. Additionally, 

the AF/SG is responsible for Coordinating with the Assistant Secretary of Defense for Health 

Affairs (ASD[HA]) on Air Force health and medical matters, and for providing guidance to 

Major Command (MAJCOM) Surgeons.37

Air Force Instruction 44-119, “Medical Quality Operations,” the central Air Force policy 

implementing DoD 6025.13, establishes policy and delegates broad oversight responsibility for 

the Quality/Process Improvement, Patient Safety, Risk Management, Professional Staff 

Management (Credentialing/Privileging) and Adverse Actions programs in the AFMS to Air 

Force Medical Operations Agency, Clinical Quality Management Division. This instruction 

outlines the roles and responsibilities of Air Force MTFs for Continuous Process Improvement 

(CPI) as implemented within the facilities. This policy requires all active duty fixed hospitals 

and freestanding ambulatory clinics to maintain accreditation by nationally recognized civilian 

agencies.38 AFI 44-119 requires that MTFs maintain ongoing self-inspection activities and 

procedures. 

36 Pursuant to 10 USC §§ 8031-8038, and as documented by paragraph 4.3 of AFMD 1, Headquarters Air Force, and 

this Headquarters Air Force (HAF) Mission Directive. Documented by: Air Force Surgeon General. Air Force 

Mission Directive 1, paragraph 4.3. 

37 Air Force Surgeon General. Air Force Mission Directive 1-48. November 2011. 

38 The Joint Commission (TJC) serves as the civilian accrediting agency for Air Force inpatient facilities. The 

Accreditation Association for Ambulatory Health Care (AAAHC) serves as the civilian accrediting agency for Air 

Force outpatient facilities.

81

 
August 29, 2014 4. Quality of Care in the Military Health System

AFMOA executes AFMS policies and strategies by engaging MAJCOMs and MTFs, whereas 

MTFs are responsible for executing AFMS policies, with oversight provided by their respective 

MAJCOM Surgeon and AFMOA. According to AFI 90-201, the Air Force Inspection System, 

Unit Effectiveness Inspections, and Wing Commander Self-Inspection Programs are regularly 

conducted to document mission readiness and compliance with Air Force policy. Additionally, 

accreditation and certification assessments by independent civilian agencies, as well as regular 

performance management forums conducted by AFMOA are used as means of monitoring MTF 

compliance with AFMS policies, accreditation standards, and performance against national 

performance benchmarks. 

Defense Health Agency/National Capital Region Medical Directorate (NCR MD) Policy Guidance

The Joint Task Force Clinical Quality Manual (JTF-CQM) 6025.01 implements DoDI 6025.13 

guidance on care quality. This manual sets procedures and responsibilities for the administration 

of the Clinical Quality Management Program by the NCR MD and describes the relationships 

between NCR MD and the Military Services for quality management. Revisions to the manual 

are managed collaboratively by the NCR MD Quality Management Department and Quality 

Working Group. 

The Director, NCR MD is responsible for the quality of health care delivered to all beneficiaries 

in the NCR MD. The Director establishes CQMP policy and serves as the governing body for 

the health care facilities. The NCR MD Quality Management department provides corporate 

guidance, monitors quality and patient safety outcomes, and collaborates with MTFs to ensure 

effective administration of the quality programs. Such programs include risk management, risk 

avoidance, safety practices, incident monitoring/management, adverse privileging/practice 

actions, sentinel events, and malpractice claims. MTF Directors ensure that hospitals maintain 

comprehensive Clinical Quality Management and Patient Safety Programs and compliance with 

the accrediting agency standards and reporting to NCR MD.

Policy Guidance for Purchased Care

Clinical quality management guidance for purchased care is found in the TRICARE Operations 

Manual (TOM) Chapter 7 (Utilization and Quality Management) and 32 CFR 199.15. DHA 

establishes the TRICARE policies that are followed and implemented by the purchased care 

contractors. The three TRICARE regional contractors (North, South, and West Regions), the six 

designated providers, and the TRICARE overseas contractor must develop a yearly clinical 

CQMP. The plan provides the framework for the contractor to objectively define and measure 

the quality of care received by beneficiaries, and is followed by an annual report. Furthermore, 

this guidance requires that contractors monitor and report National Quality Forum (NQF) Serious 

Reportable Events (SREs) and the Agency for Healthcare Research and Quality (AHRQ) Patient 

Safety Indicators.

The CQMP plan must address several key elements, including the organization’s structure and 

staffing and qualifications for quality management staff. It must fully address the quality review 

process, to include how grievances and potential quality issues are investigated and resolved. 

Also, it must describe how quality improvement and patient safety initiatives are selected and 

82
Military Health System Review – Final Report August 29, 2014

monitored, and the measureable objectives it will use for internal monitoring and improvement 

of its clinical quality program. 

Annual CQMP plans and reports are monitored and reviewed by the quality management staff 

from the TRICARE Regional or Area Offices and DHA to ensure compliance with contract 

requirements, and ensure high-quality and safe care is being provided to TRICARE beneficiaries 

in the purchased care network.

Review of Internal and External Studies on Quality of Care 

The MHS Review Group identified 51 studies and reports on MHS care quality, which included 

studies designed and conducted by DHA and those conducted by independent organizations. Of 

the studies identified, 28 were found to be pertinent and within scope of this review. A 

discussion of findings and summaries of reports examined can be found in Appendix 4.2. Key 

findings from the review of these reports include: 

1. The lack of clinical quality and outcome data on care rendered in the purchased care 

component has been identified as a significant gap. Reporting from civilian facilities and 

individual providers to the Government is voluntary. The current DHA/contracting 

reimbursement methodology does not provide the flexibility required to incorporate 

quality performance reporting into reimbursement rate negotiations for contractors. 

2. New MHS governance provides a better forum for collaborative work among the 

components on quality efforts.

3. The MHS lacks a defined process for communicating relevant study findings and 

recommendations and for tracking programs and outcomes across the enterprise.

4. Gaps were identified in the MHS data system’s ability to support efficient, bidirectional 

transmission of data between inpatient and outpatient electronic records.

 Recommendation for Responding to Prior Reviews of MHS Quality

a. DHA should integrate requirements for purchased care clinical quality data on 

TRICARE beneficiaries into the TRICARE Operations Manual and future 

TRICARE regional contracts.

Gap Analysis

DoDI 6025.13, Medical Quality Assurance (MQA) and Clinical Quality Management in the 

Military Health System (MHS) and its supporting manual, DoDM 6025.13, are focused on 

credentialing, privileging, risk management, and patient safety but lack specificity regarding 

quality measurement and process improvement. 

1. There are gaps in the enterprise process to validate that the Services and MTFs are 

compliant with the implementation of policies and directives disseminated from 

ASD(HA). An example of this gap is the inconsistent and incomplete implementation of 

HA Policy 11-003, Policy for Comprehensive Pain Management. The new MHS 

governance system can work effectively to standardize policy implementation across 

83
August 29, 2014 4. Quality of Care in the Military Health System

DoD; the variation in Comprehensive Pain Management has been identified by the MOG, 

which has consolidated the working groups focused on pain management to implement a 

DoD-wide policy.

2. There is disparity in how the Services monitor and document policy compliance, with no 

clear guidance in DoD and Service-level policies. The differences among Services (other 

than accreditation and certification by civilian agencies) contribute to difficulty in DHA 

oversight of quality in the direct care component. The Army has established a Clinical 

Assurance Performance Directorate that reports to the SG to ensure standardization in all 

MTFs. Previously, the Command chain monitored policy compliance, and this new 

Directorate will serve as an enforcement agency for policy. The Navy utilizes trained 

TJC Fellows to survey each MTF in their area of responsibility in between the TJC 

triennial survey (and as needed) to assess compliance with standards. In the Air Force, 

MTF compliance with policies is measured by military inspections (Unit Effectiveness 

Inspections) and by regular Performance Management Forums with AFMOA. The new 

Air Force Inspection System, with revised Air Force Instructions and Self-Inspection 

Checklists will reduce variability and clarify policy guidance, put compliance oversight 

in the hands of commanders, and refocus the biennial Unit Effectiveness Inspections. 

3. The third significant gap relates to lack of clear policy on required education and training 

to optimally prepare personnel at all levels in quality management. While each Service 

has developed policy and/or programs related to education and training in quality, this is 

not directed from HA or DHA.

 Recommendations to Address Gaps in Training and Compliance with Policies

a. ASD (HA) and DHA should develop policy guidance in support of DoDI and DoDM 

6025.13 with specific direction on quality measurement, performance improvement, 

and requirements for education and training.

b. ASD (HA) should develop policy guidance to manage and track compliance of the 

Services and DHA with applicable DoD policies and directives. 

Purchased Care Gap Analysis

In American medicine, primary responsibility for quality of care rests with providers, not 

insurance carriers or government programs that reimburse those providers. Nonetheless, policies 

and procedures governing the reimbursement programs should reinforce those provider 

responsibilities. The lack of clinical quality and outcome data on care rendered in the purchased 

care component is a significant gap. Reporting from civilian facilities and individual providers 

to the Government is voluntary. Current best business practice occurs when payers (health 

insurance companies) have the flexibility to negotiate reimbursement rates that include quality 

data reporting. The Centers for Medicare & Medicaid Services (CMS) (Advantage Programs) 

and Veterans Affairs (civilian sector contracts) have introduced changes in their reimbursement 

rates that include clinical quality data reporting. 

84
Military Health System Review – Final Report August 29, 2014

Education and Training

All Service components and the purchased care sector conduct a wide variety of training on 

quality and performance improvement. The complete assessment of education and training 

efforts is located in Appendix 4.3. 

Not unexpectedly, variation among the Services exists in the conduct and monitoring of training. 

Services vary in their visibility of training completion. The Services have invested significantly 

in Lean/Six Sigma performance improvement training to provide the necessary skills to drive 

performance improvement throughout the system. The MHS Review Group concluded the 

following regarding training related to care quality:

1. Although there is quality training occurring in the Services, there is no clearly prescribed 

DoD policy specific to quality training and education.

2. There is no clearly defined quality of care career development pathway.

3. There has been a significant investment in Lean/Six Sigma performance improvement 

training. 

 Recommendations Regarding Quality of Care Training

a. The DHA Education and Training Directorate should conduct an in-depth review 

and needs assessment of quality training to adequately assess the efficacy of 

training.

b. MHS governance should determine the requirements to guide the development and 

implementation of a quality expert career path. 

Data Analysis

The measures of quality care used for this analysis were selected to support alignment of the 

MHS with the goals and philosophies of the Institute of Medicine, the Institute for Healthcare 

Improvement, National Quality Forum (NQF), and the Agency for Healthcare Research and 

Quality (AHRQ). The criteria for selecting performance measures and respective data sources 

include:

• Health care quality measures were selected based on their acceptance and use by the 

national health care community, the domains of quality addressed (safety, timeliness, 

effectiveness, efficiency, equity, and patient-centered), and the availability of the defined 

data elements.

• Performance measures were required to have defined national benchmarks, whenever 

possible.

• Data sources were used if they provided quality of care information for either direct care, 

purchased care, or both.

• Data sources were selected to provide information on the satisfaction and perception of 

patients regarding MHS quality of care.

85
August 29, 2014 4. Quality of Care in the Military Health System

The measures of quality presented in this report include:

Accreditations and Certifications: Accreditation and certifications reflect whether MHS 

systems have met nationally recognized guidelines and requirements for quality, safety, and 

uniform processes.

Healthcare Effectiveness Data and Information Set (HEDIS®): The MHS compares its 

performance in selected measures against national HEDIS® benchmarks for this measure set,

which is utilized by more than 90 percent of health plans in the United States.

Quality of Care in the Purchased Care Component Using Hospital Compare: Hospital 

Compare is a consumer-oriented website that contains information about the quality of care 

at more than 4,000 Medicare-certified hospitals in the United States. The MHS tracks 

several process-of-care measures common to the direct care and purchased care components. 

ORYX National Hospital Quality Measures: ORYX is a set of measures used by TJC in 

its hospital accreditation process, in which all military hospitals participate. 

Prevention Quality Indicators (PQI): The Prevention Quality Indicators (PQI) are 

measures of potentially avoidable hospitalizations for ambulatory care sensitive conditions; 

they are an indicator for the quality of ambulatory care provided.

Thirty-Day Readmissions: Hospital readmissions within 30 days may occur due to an 

unrelated diagnosis or a planned course of treatment, or may reflect incomplete care, or a 

complication of care during the initial admission. 

National Perinatal Information Center (NPIC): The National Perinatal Information 

Center (NPIC) utilizes MHS direct care data to compare the quality of care provided to 

pregnant women and newborns against averages of metrics derived from data submitted by 

86 participating hospitals. Because many perinatal metrics do not have national benchmarks, 

the MHS uses NPIC averages for comparison.

National Surgical Quality Improvement Program (NSQIP): NSQIP is a voluntarily 

reported, data-driven, case-mix adjusted, risk-adjusted, outcome-based program developed 

by the American College of Surgeons to measure and improve the quality of surgical care. 

Inpatient Mortality Measures: The quality of inpatient care is measured using the AHRQ 

developed Inpatient Quality Indicator (IQI) measure set, which contains a number of 

condition-specific mortality measures. In addition, risk-adjusted mortality ratios were 

calculated for all inpatient MTFs based on a commonly accepted (Elixhauser) methodology. 

Experience of Care: The experience of care measures use survey data to determine 

beneficiary satisfaction with MHS health care. AHRQ CAHPS and Hospital Consumer 

Assessment of Healthcare Providers and Systems (HCAHPS) comparable questions were 

used to allow comparison to civilian systems and benchmarks.

86
Military Health System Review – Final Report August 29, 2014

Accreditation and Certification

Accreditation and certification serve as a formal declaration that programs and personnel within 

the MHS have met standardized guidelines and requirements relative to quality, safety, and 

uniform quality management system processes. Further, they demonstrate the commitment of 

the MHS to providing the highest quality of care. For the purposes of this report, the Services 

provided accreditation and certification information for their MTFs in a number of categories. 

Categories included accreditation and certification information on health care services, such as 

primary care, laboratory and blood banks, radiology and nuclear medicine, medical and surgical 

subspecialties (oncology, hyperbaric medicine, and trauma care), and advanced medical and

dental education.

MHS-Level Discussion

All MTFs that sought TJC accreditation obtained it. In 2013, of the civilian facilities that sought 

TJC accreditation in the United States overall, 7.3 percent of ambulatory programs, 5.1 percent 

of home care, and 4.8 percent of hospitals were not accredited. All standalone ambulatory health 

centers within the direct care component are accredited either by TJC or the Accreditation 

Association for Ambulatory Heath Care (AAAHC). In addition, direct care facilities participate 

in more than 7 laboratory and blood bank certification programs, more than 6 radiology and 

nuclear medicine certification programs, more than 11 subspecialty certification programs, and 

more than 20 advanced medical and dental education recognition programs.39 All MTF-based 

laboratories are inspected by the College of American Pathologists (CAP) and 100 percent are 

accredited. Hospitals and laboratories in the purchased care component are contractually 

required to be inspected and accredited by a CMS-recognized accreditation body.

Service-Level Discussion

For the Patient Centered Medical Home (PCMH) Initiative, 76 percent of the Army facilities, 

100 percent of the Navy facilities, 87 percent of the Air Force facilities, and both NCR MD 

facilities have received at least one National Committee for Quality Assurance (NCQA) or 

AAAHC recognition (data not shown). Only 10 percent of U.S. primary care practices, close to 

7,000 in total, are recognized as PCMHs by NCQA. Accreditations and certifications by type 

and service are displayed in Appendix Table 4.4-1. Variations among the Services can be 

explained by differing requirements and policies. The services continue to work on obtaining 

national recognition to obtain PCMH practice. This continues to vary due to service specific 

timelines as goals for completion.

39 Several facilities reported participating in certification programs but were not specific in the type of program; 

hence the use of the term “more than n programs.”

87

 
August 29, 2014 4. Quality of Care in the Military Health System

Facility Type, Location, and Other Discussions

Results indicate that there are no substantial variations in accreditations and certifications by 

geographic location with OCONUS MTFs accredited by U.S. certification organizations. 

Additional findings include:

1. One hundred percent of the MTFs are accredited. However, it is noted that aggregation 

and analysis of accreditation findings is not currently shared across the MHS. The 

Services currently have education and training with industry programs that allow officers 

to complete a fellowship with TJC. Post training utilization of these fellows varies by 

Service.

 Recommendations Regarding Accreditation and Certification

a. MHS governance should establish a mechanism to aggregate and communicate 

accreditation findings across the MHS.

b. MHS governance should evaluate the utility of adding additional fellowship 

opportunities with TJC or other nationally recognized programs, and the Services 

should explore optimizing and standardizing Service fellow utilization by aligning 

training with follow-on assignment after fellowship completion. 

HEDIS® Measures of Performance

NCQA developed and maintains HEDIS®, a tool used by more than 90 percent of U.S. health 

plans to measure performance on important dimensions of care and service using 81 measures 

across 5 domains of care. HEDIS® allows consumers to compare an individual health plan’s 

performance to other plans and to national benchmarks. HEDIS® measures are reported as the 

percentage of eligible patients receiving a service and then compared to the national NCQA 

benchmark percentiles. In this report, average national Health Maintenance Organization 

(HMO) performance data for 2012 are also provided for comparison purposes.40 A comparison 

of HEDIS® results by MTF facility type (medical center compared to community hospital) was 

not performed. 

The MHS compares its performance in selected measures against the national HEDIS® 

benchmarks (see Appendix Table 4.4-2).41 DHA adheres to the specifications to collect and 

calculate performance for each measure, and is audited annually by a NCQA-certified HEDIS®

auditor for compliance.

40 National Committee for Quality Assurance. (2013). Improving Quality and Patient Experience. The State of 

Health Care Quality.

41 Data obtained from the most recent year as published in: Annual Quality Compass, 2014. National Committee 

for Quality Assurance. Available at: http://store.ncqa.org/index.php/data-and-reports/quality-compass-2014.html. 

Accessed July 25, 2014.

88

 
Military Health System Review – Final Report August 29, 2014

The MHS, through the Clinical Measures Steering Panel, identifies which HEDIS® measures to 

review based on data availability42 and their relevance to the health plan, the Services, and 

MTFs. The MHS has selected 18 HEDIS® measures to review in the direct care component, and 

12 for review in the purchased care component.43 If the MHS consistently performs above the 

90th percentile on a given measure over a period of several years, routine data collection for that 

measure may be discontinued, and the measure replaced with a new one. Performance on this 

“retired” measure is examined periodically to ensure continued success. 

MHS-Level Discussion: In comparing the performance of the MHS across the 18 selected

HEDIS

® measures with national benchmarks, only one measure—comprehensive diabetes care, 

Glycosylated Hemoglobin (HbA1c) <7 percent for a selected population—outperformed the 

NCQA 90th percentile benchmark, the stated MHS goal for all HEDIS measures. Five of the 

measures were between the 75th and 89th percentile of their NCQA benchmarks. Three 

measures were below the NCQA 25th percentile: cholesterol management for patients with 

cardiovascular conditions (LDL-C screening); comprehensive diabetes care (HbA1C screening); 

and comprehensive diabetes care (LDL cholesterol screening). While many MTFs fall below the 

50th percentile for certain measures, they actually exceed the national average for performance. 

In addition, it is important to consider that in reviewing HEDIS® data, there may only be a few 

percentage points of performance that separate a facility from the next higher or lower quartile. 

Of note, between 2012 and 2013, the MHS demonstrated statistically significant improvements 

on 10 of the 18 measures, while there were statistically significant declines in performance on 6 

HEDIS® measures (see Appendix Table 4.4-3). 

Among the measures below the NCQA 50th percentile benchmark in 2013, several improved 

between 2012 and 2013: the management of antidepressant medication for the continuation 

phase; LDL-C control for patients with cardiovascular conditions; diabetic HbA1C screening; 

diabetic LDL screening; and well-child visits.

Service-Level Discussion: Navy MTFs led the MHS in the number of HEDIS® measures above 

the 75th percentile (12 of 18) with 72 percent of measures showing significant improvement 

from 2012 to 2013 (data not shown). Army MTFs closely followed with 10 of 18 above the 75th 

percentile, 9 of which showed significant improvement from 2012 to 2013. Among Air Force 

MTFs, 7 measures were above the 75th percentile, with only 27 percent of the measures 

improving statistically from 2012 to 2013. In the two National Capital Region MTFs, 8 

measures were above the 75th percentile NCQA benchmarks (year-over-year changes in 

performance were not calculated because only 2013 data were available) (see Appendix Table 

4.4-4). Given the distribution of the results among the Services, there may be Service-specific

processes and practices that account for some of the differences in the HEDIS® rates. 

42 All data are currently obtained administratively through a hybrid methodology that involves chart abstraction.

43 The six exclusions for purchased care are due to the inability to obtain laboratory values from claims data required 

to calculate these measures.

89
August 29, 2014 4. Quality of Care in the Military Health System

Location: Notable differences were found on HEDIS® measure performance among MTFs 

located in the continental United States (CONUS) and those outside (OCONUS). Among 

OCONUS MTFs, 9 of the 18 measures were above NCQA’s 75th percentile, in contrast to 6 

among CONUS facilities (see Appendix Table 4.4-5). It should be pointed out that overseas 

facilities only enroll active duty Service members and their families, which may help explain 

why overall performance appears to be better than in CONUS facilities. In addition, most 

overseas facilities are small, there is less opportunity to use other health insurance with civilian 

providers to obtain preventive services, and providers may have greater opportunities to ensure 

their patients receive preventive services in a timely manner (Appendix Table 4.4-7 and 

Appendix Table 4.4-8). 

Purchased Care: The purchased care component only monitors 12 HEDIS® measures due to 

the unavailability of required clinical data elements. Appendix Table 4.4-6 shows purchased 

care HEDIS® measures as compared to NCQA national benchmarks and the HMO national 

average for 2012, with 7 of the 12 measures monitored falling below the NCQA 25th percentile, 

as well as the 2013 national HMO average. Four of the 12 monitored measures were above the 

50th, but below the 75th NCQA percentile. There are mitigating factors that can account for 

some of the considerable lag between HEDIS® measure performance in the purchased care 

component compared to direct care (see Appendix 4.4: HEDIS® Methodological 

Considerations). 

Comparison to External Health Systems: HEDIS® data for the MHS were compared to data 

from three external health plans for the same period (2013). In the MHS, data are collected from 

both direct and purchased care components for 12 measures; for 6 additional measures, data are 

collected from the direct care component only (chlamydia screening, cholesterol management, 

diabetes <=9, <7, <8, diabetes LDL Control) (Table 4.2). There are inaccuracies in the data 

collected from the purchased care component due in part to the inability to capture clinical data 

that may be lacking in the MHS Population Health Portal, and the inability to accurately exclude 

beneficiaries that may have other health insurance from rate calculations. Data were not 

available from Health System 2 for 7 measures (asthma medication and the six components of 

comprehensive diabetes care). Of the available data, the MHS performed above the benchmark 

percentile in 8 instances, at the same benchmark percentile in 11 instances, and below the 

benchmark percentile in 28 instances. 

90
Military Health System Review – Final Report August 29, 2014

Table 4.2 Comparison of HEDIS® 2013 Data for MHS Against External Comparison Organizations 

(Health Systems 1, 2, and 3)

HEDIS® Measures 

MHS 2013 

HS 1 2013 

HS 2 2013 HS 3 2013 MHS 2013 

Percentile 

HS 1 2013 

Percentile 

HS 2 2013 

Percentile 

HS 3 2013 

Percentile 

Antidepressant Medication Management: 

Acute Phase 68.51 72.73% 76.16 63.59 50th 75th 90TH 25th

Antidepressant Medication Management: 

Continuation Phase 46.08 54.20% 67.67 47.55 25th 75th 90TH 25th

Use of Appropriate Medications for 

People With Asthma: Asthma Medication 

Rate

94.71 94.01% NA 94.33 75th 75th NA 75th

Breast Cancer Screening 68.88 79.05% 79.39 70.01 25th 90th 90th 25th

Cervical Cancer Screening 76.87 86.75% 82.15 76.83 25th 90th 75th 25th

Chlamydia Screening in Women: Total 

Rate 59.72 71.14% 47.73 28.86 75th 90th 50th < 25th 

Cholesterol Management for Patients 

with Cardiovascular Conditions: LDL 

Control (<100 mg/dL)

59.57 76.72% 73.25 68.19 25th 90th 90th 75th 

Cholesterol Management for Patients 

with Cardiovascular Conditions: LDL 

Cholesterol Screening

77.56 95.64% 93.42 86.52 < 25th 90th 75th 25th 

Colorectal Cancer Screening 69.82 75.77% 75.59 70.98 75th 90th 90th 75th 

Comprehensive Diabetes Care: Poor 

Glycemic Control (HbA1c >9 percent)— 

Lower rates signify better performance

78.2 78.86% NA 77.55 50th 75th NA 50th 

Comprehensive Diabetes Care: HbA1c 

<7 percent for a Selected Population 53.53 41.05% NA 45.41 90th 25th NA 50th 

Comprehensive Diabetes Care: Good 

Glycemic Control (HbA1c <8 percent) 70.38 66.59% NA 64.78 75th 50th NA 50th

Comprehensive Diabetes Care: HbA1c 

Screening 84.89 94.13% NA 92.7 < 25th 75th NA 50th 

Comprehensive Diabetes Care: LDL 

Cholesterol Control (<100 mg/dL) 55.8 63.61% NA 55.66 75th 90th NA 75th 

Comprehensive Diabetes Care: LDL 

Cholesterol Screening 80.69 92.01% NA 81.75 < 25th 90th NA < 25th 

Follow-Up After Hospitalization for Mental 

Illness: Within 30 Days Post-Discharge 74.84 84.44% 85.87 71.55 25th 75th 75th 25th 

Follow-Up After Hospitalization for Mental 

Illness: Within 7 Days Post-Discharge 58.46 75.26% 75 48.75 25th 75th 75th < 25th 

Well-Child Visits (Ages 0–15 Months): Six 

or More Well-Child Visits 79.15 84.45% 96.98 79.57 25th 50th 90TH 25th 

2014 MHS Review Group

Source: HEDIS - MHS Population Health Portal, June 2014

Findings related to MHS performance on HEDIS® measures include: 

1. Purchased care data show that 7 of the 12 HEDIS® measures monitored fall below the 

NCQA 25th percentile benchmark, and 4 were in the 50th percentile. The measures 

identified for improvement include breast and cervical cancer screening, cholesterol 

screening, comprehensive diabetes care (Hb A1c screening and LDL-cholesterol 

screening), and mental health follow up. 

Indicates 90th percentile Indicates 75th percentile Indicates 50th percentile Indicates 25th percentile Indicates lower than 25th percentile

91 
August 29, 2014 4. Quality of Care in the Military Health System

2. The MHS collects data on 18 HEDIS® measures for direct care and 12 measures for 

purchased care. NCQA has 81 HEDIS® measures. Direct care outperformed the NCQA 

90th percentile benchmark for 1 HEDIS® measure. Five of the direct care HEDIS® 

measures were between the 75th and 89th percentile of the NCQA benchmarks. Two 

measures are at the 50th percentile while 7 are between the 25th and 50th percentile. 

Three measures were below the NCQA 25th percentile. When comparing OCONUS to 

CONUS MTFs in 2013, overseas facilities performed better on nine measures and 

similarly on five measures.

3. Between 2012 and 2013, the MHS demonstrated statistically significant improvements on 

10 of the 18 measures, while there were statistically significant declines in performance 

on 6 HEDIS® measures. 

 Recommendations Related to MHS Performance on HEDIS® Measures

a. DHA Health Plans should give purchased care contractors the authority to use 

supplemental databases to improve the capture of clinical information for 

purchased care enrollees.

b. DHA Health plans should evaluate alternative methods of incentivizing contractors 

and/or providers to improve the provision of clinical preventive services and 

HEDIS® performance. This may require statutory or regulatory changes, since 

new, innovative payment mechanisms may have to be developed to encourage 

compliance.

c. MHS governance should assess the value of expanding the number of HEDIS® 

measures monitored to evaluate care provided to enrolled beneficiaries.

d. MHS governance should establish policy to guide processes for verification of 

clinical data and capture in AHLTA (DoD’s outpatient electronic health record)

regarding preventive services that are obtained outside of the direct care 

component. 

e. DHA should develop plans to improve Other Health Insurance documentation in 

DEERS for all beneficiaries to ensure those with Other Health Insurance are not 

included in HEDIS® calculations.

f. MHS governance should develop a strategy for MTFs to maximize the use of “action 

lists” generated by the MHS Population Health Portal to ensure beneficiaries 

receive clinical preventive services in a timely manner. 

Quality of Care in the Purchased Care Component 

TRICARE contractors provide quality of care oversight for services provided to beneficiaries in 

the TRICARE network. This activity includes credentialing of network providers, validating 

accreditation status of participating facilities, and addressing quality of care concerns through 

established procedures. Hospital accreditation is a requirement (32 C.F.R.§ 199.6) to become a 

TRICARE-authorized provider and is also required for network credentialing. Due to disparate 

information systems used by network providers, accurate measurement of the quality of care 

provided in the purchased care component is not possible. However, information provided by 

network facilities on websites such as Hospital Compare serves as an excellent surrogate for the 

quality of care received by TRICARE beneficiaries at those facilities. 

92 
Military Health System Review – Final Report August 29, 2014

Hospital Compare is a consumer-oriented website that contains information about the quality of 

care at more than 4,000 Medicare-certified hospitals in the United States. Established by CMS 

and the Hospital Quality Alliance, the website supports the improvement of hospitals’ quality of 

care and assists consumers in making informed decisions about their health care.44 Hospital 

Compare contains information regarding the timeliness and effectiveness of care, as well as 

information on readmissions, complications, and deaths. The vast majority of TRICARE 

network hospitals participate in the Hospital Compare program.

Each of the TRICARE Regional Offices uses Hospital Compare data in its performance of 

quality oversight of purchased care. This information provides MTFs with baseline knowledge 

of the quality of care provided by network hospitals in its area of service. The MHS tracks 

several care measures common to the direct care and purchased care components that are 

highlighted on the Hospital Compare website. The measures monitored for both direct and 

purchased care are displayed in Table 4.3.

Table 4.3 National Hospital Quality Measures Monitored for Both Direct and Purchased Care

Measures

Acute Myocardial Infarction (AMI)

Heart Failure (HF)

Pneumonia (PN)

Surgical Care Improvement Project (SCIP)

Children's Asthma Care (CAC)

2014 MHS Review Group

Source: DoD Joint Commission and Hospital Compare Database, June 2014

Hospitals reporting on Hospital Compare do not differentiate TRICARE beneficiaries from other 

patients. MHS National Hospital Quality Measures - Hospital Compare are reviewed annually 

by the Services and DHA staff at the Quarterly Clinical Measures Steering Panel meeting.

MHS-Level Discussion: A review of the Hospital Compare data reveals that TRICARE 

network hospitals are performing at or above the national benchmarks on a composite measure of 

Hospital Compare performance (see Figure 4.1 and Appendix Figures 4.4-1) 

44 Medicare Hospital Compare. Available at: www.hospitalcompare.hhs.gov. Accessed July 25, 2014.

93

 
August 29, 2014 4. Quality of Care in the Military Health System

Figure 4.1 Purchased Care Compliance Rate, FY09 – FY1245

2014 MHS Review Group

Source: MHS MART (M2) Comprehensive Ambulatory/Professional Encounter Record (CAPER), July 2014

 Recommendation Regarding Quality Data in the Civilian Network 

a. DHA should integrate requirements for purchased care clinical quality data on 

TRICARE beneficiaries into the TRICARE Operations Manual and future 

TRICARE regional contracts.

ORYX® – National Hospital Quality Measures

The Joint Commission (TJC) is the accrediting agency for military hospitals; Table 4.4 shows the 

core measures of hospital quality used to meet TJC accreditation requirements.

45 Note: DP indicates Designated Providers/US Family Health Plan.

86

88

90

92

94

96

98

100

2009 2010 2011 2012

Rate of Compliance

FY

National

MCSC

TRO_West

TRO_North

TRO_South

DP

 

 

 

94

 
Military Health System Review – Final Report August 29, 2014

Table 4.4 TJC Core Measure Sets Monitored for both Direct and Purchased Care

Acute Myocardial Infarction (AMI) Hospital Outpatient (OP)

Heart Failure (HF) Stroke (STK)

Pneumonia (PN) Venous Thromboembolism (VTE)

Surgical Care Improvement Project (SCIP) Immunization (IMM)*

Children’s Asthma Care (CAC) Substance Use (SUB)*

Perinatal Care (PC) Tobacco Treatment (TOB)*

Hospital Based Inpatient Psychiatric Services (HBIPS) Emergency Department (ED)*46

2014 MHS Review Group

Prior to January 2014, TJC required accredited hospitals to submit data on four core measure sets 

each quarter. Currently, TJC requires data on six measures to be submitted.47 All accredited 

hospitals with one or more patients in the measure population must submit data on acute 

myocardial infarction (AMI), heart failure (HF), perinatal care (PC), and Surgical Care 

Improvement Project (SCIP) measures. Accredited hospitals with 1,100 or more annual births 

must also submit PC data. Facilities select one or two additional measure sets to complete the 

six core measure set requirement. A number of smaller military hospitals may be required to 

select as many as four additional measures sets to meet the requirement.

CMS National Hospital Quality measures (with similar specifications to TJC measures) are used 

to evaluate the process of care in TRICARE network hospitals (purchased care). DHA selected 

the measures of AMI, PN, SCIP, and children’s asthma care (CAC) for Service review to align 

with data available on CMS Hospital Compare. Two additional measures were also selected for 

abstraction across the Services: PC was selected due to the high volume of births in military 

hospitals, and hospital-based inpatient psychiatric services (HBIPS) was selected because these 

patients represent high-risk populations.

Table 4.5 shows the composite results for 13 ORYX® measures, with direct care having a lower 

rate for 9 of the measures but outperforming in PC and venous thromboembolism (VTE)

measures. 

46 Asterisk indicates measures introduced in 2012, therefore only two years of data were available for analysis.

47 Core Measure Sets. The Joint Commission. http://www.jointcommission.org/core_measure_sets.aspx. 

Accessed: July 25, 2014. 

95

 
August 29, 2014 4. Quality of Care in the Military Health System

Table 4.5 Composite Measures for 13 ORYX Measures

Composite Measure Set National Purchased Care Direct Care

Heart attack care (AMI) 98.97% 95.30% 97.03%

Children's asthma care (CAC) 95.97% 98.80% 87.37%

Heart failure (HF) 97.49% 96.70% 94.79%

Inpatient psychiatric services (HBIPS) 91.56% ND 87.08%

Hospital outpatient department (OP) 97.84% ND 85.41%

Perinatal care (PC) 63.58% ND 74.30%

Pneumonia care (PN) 97.54% 96.90% 94.30%

Surgical care (SCIP) 98.64% 98.00% 97.50%

Stroke care (STK) 96.75% ND 96.04%

Venous thromboembolism care (VTE) 92.28% ND 94.77%

Immunization (IMM) 91.17% ND 74.01%

Substance use (SUB) 60.13% ND 43.26%

Tobacco treatment (TOB) 88.17% ND 85.52%

2014 MHS Review Group; ND indicates “No Data”

Source: DoD Joint Commission Core Measure Database, June 2014

A deeper analysis of the results shows the direct care component was significantly below the 

national average for the following measures over most of the 2010 to 2013 period, which 

contributes to low composite measures: AMI-8a (primary percutaneous coronary intervention), 

CAC3 (home management plan of care given to patient/caregiver), HF1 (discharge instructions), 

PN3b (blood cultures performed in the ED prior to initial antibiotic in hospital), SCIP2a 

(prophylactic antibiotic selection for surgical patients), SCIP Card-2 (surgery patients on betablocker therapy prior to arrival who received a beta-blocker during the perioperative period), 

SCIP VTE-2 (surgery patients who received appropriate venous thromboembolism prophylaxis 

within 24 hours prior to surgery to 24 hours after surgery), IMM 1a (pneumococcal 

immunization rate), and IMM 2 (influenza immunization) (see Appendix Figures 4.4-2).

Purchased care rates are for FY 2012 and include all patients serviced at the network civilian 

hospitals, not just patients enrolled to TRICARE.

Index scores based on TJC calculations for control limits are used to evaluate whether an MTF’s 

process of care is stable (in statistical control) because only common cause variation exists or an 

MTF’s process of care is unstable (out of statistical control) because special cause variation 

exists (see Appendix Table 4.4-9a and Appendix Table 4.4-9b for additional information). 

Direct Care Performance: As a system, the direct care component is excelling in 16 measures, 

showing improvement or meeting target levels on 26 other individual measures, and requiring 

improvement on 18 measures. (See Appendix Table 4.4-9c, which presents the direct care scores 

for individual measures that made up each set of core measures over the period, 2010 to 2013. 

96
Military Health System Review – Final Report August 29, 2014

For example, the direct care component obtained perfect, or significantly better, scores for all but 

one of the stroke measures.)

The most recent year of performance shows that 23 of the 55 hospitals included in this analysis 

(42 percent) are meeting or exceeding TJC requirements on all composite measures. (See 

Appendix Table 4.4-10 for description of measures). Eight of the 55 hospitals (15 percent) need 

improvement on one composite measure. The remaining 24 of 55 hospitals (43 percent) need 

improvement on two or more composite measures (Appendix Table 4.4-11).

Service Level, Facility Type, Location, and Other Discussions: There are 55 direct care 

facilities included in this analysis. The breakdown by Service, facility type, and geographic 

location is as follows:

• 22 Army, 18 Navy, 13 Air Force, 2 DHA

• 15 Medical Centers, 40 Hospitals

• 41 CONUS, 14 OCONUS

At a minimum, facilities must maintain a composite rate of 85 percent on accountability 

measures to meet TJC compliance. All facilities, (100 percent) meet this requirement. Facilities 

are considered Top Performers when they maintain a composite rate of 95 percent (see Appendix 

Table 4.4-12).

Comparison to External Health Systems: When direct care rates were compared to HS1, HS2, 

and HS3 rates for AMI, HF, PN and SCIP, the direct care component of the MHS was found to 

have the lowest rates on 17 of 20 measures. In most instances, direct care rates were within five 

percentage points of the other health organizations. Measures showing the largest differences 

were AMI 7a (50 percentage points below), AMI 8a (36 to 43 percentage points below), HF 1 (7 

to 10 percentage points below), PN 6a (12 to 22 percentage points below), and SCIP Card 2 (8 to 

10 percentage points below). HS1 is clearly a top performer, consistently maintaining a rate of 

100 percent across several measures. The direct care component is outperforming HS2 and HS3

on PC and VTE measures (see Appendix Table 4.4-13).

In summary:

• During the entire reporting period, all MTFs achieved the mandatory composite 

performance rate of at least 85 percent on ORYX accountability measures. Two MTFs 

were TJC Top Performers in 2010; four in 2011 and four in 2012. Most of these were in 

the Core Measure areas of SCIP and VTE. The MHS is consistently falling significantly 

below the national average on nine measures.

 Recommendations Regarding MHS Performance on National Hospital Quality 

Measures

a. DHA Health Information Technology should prioritize electronic health record 

upgrades by aligning needed data elements into Essentris (the inpatient electronic 

97
August 29, 2014 4. Quality of Care in the Military Health System

health record). All inpatient MTFs should have the capability to remotely access 

health records to facilitate expeditious and timely data extraction for clinical 

measure calculation.

b. MHS governance should establish goals and processes for increasing the number of 

MTFs achieving The Joint Commission Top Performer status annually.

Prevention Quality Indicators (PQI) 

AHRQ develops and maintains PQIs, which are measures of potentially avoidable 

hospitalizations for ambulatory care sensitive conditions. These are conditions for which good 

outpatient care can potentially prevent the need for hospitalization or for which early 

intervention can prevent complications or more severe disease. The PQIs are population-based 

measures that also account for important hospitalization-related factors. Even though these 

indicators are based on hospital inpatient data, they provide insight into the quality of outpatient 

health care services. PQIs include the indicator measures presented in Table 4.6. 

Table 4.6 PQI Indicator Measures

PQI Measures

PQI 01 Diabetes Short-Term Complications Admission Rate

PQI 02 Perforated Appendix Admission Rate

PQI 03 Diabetes Long-Term Complications Admissions Rate

PQI 05 COPD or Asthma in Older Adults Admissions Rate

PQI 07 Hypertension Admission Rate

PQI 08 Congestive Heart Failure Admission Rate

PQI 09 Low Birth Weight

PQI 10 Dehydration Admission Rate

PQI 11 Bacterial Pneumonia Admission Rate

PQI 12 Urinary Infections Admission Rate

PQI 13 Angina without Procedure Admissions Rate

PQI 14 Uncontrolled Diabetes Admissions Rate

PQI 15 Asthma in Younger Adult Admission Rate

PQI 16 Lower Extremity Amputations (AMP) Among Patients with Diabetes Rate

PQI 90 Overall PQI Composite

2014 MHS Review Group

Source: Military Health System Mart (M2), July 2014

MHS and Service-Level Discussion: The PQIs are a relatively new set of measures in the 

direct care component. The Services have had minimal exposure to these measures and have not 

integrated them into ongoing quality programs. In addition, the MHS lacks a policy that governs 

the use of the PQI indicators. While PQI data have not been reviewed by the Services, DHA has 

collected and reviewed these data and preliminary conclusions have been drawn. 

98
Military Health System Review – Final Report August 29, 2014

A review of MTF performance on the PQI measures from 2010 to 2013 demonstrates overall 

good performance with 89 percent of direct care PQI measures meeting or exceeding the national 

AHRQ benchmarks. Service-level compliance rates for specific measures are outlined in Table 

4.7. (Note: Data not available for PQI 09 - Low Birth Weight.) 

At the Service level for the same time period, NCR MD and Navy demonstrated 90 to 92 percent 

of their measures better than the AHRQ benchmarks, and the Army and Air Force achieved 87 to 

89 percent of their measures better than the AHRQ benchmarks. The only measure with 

significantly low performance across all services was PQI 02-Perforated Appendix Admission 

Rate.

Table 4.7 PQI Compliance Rate by Service (2010 – 2013)

PQI Indicator

Air 

Force Army Navy NCR

PQI 01 - Diabetes Short-Term Complications Admission Rate 92.57 87.37 91.72 91.53

PQI 02 - Perforated Appendix Admission Rate 63.86 65.61 67.72 64.10

PQI 03 - Diabetes Long-Term Complications Admission Rate 94.75 92.49 95.89 100.00

PQI 05 - COPD or Asthma in Older Adults Admission Rate 99.31 98.14 99.00 100.00

PQI 07 - Hypertension Admission Rate 77.01 78.99 87.49 74.58

PQI 08 - Congestive Heart Failure Admission Rate 95.34 94.15 95.18 100.00

PQI 10 - Dehydration Admission Rate 73.44 73.34 84.19 77.97

PQI 11 - Bacterial Pneumonia Admission rate 86.52 79.92 87.08 100.00

PQI 12 - Urinary Infections Admission Rate 81.96 77.19 85.72 86.44

PQI 13 - Angina without Procedure Admission Rate 84.34 79.45 90.37 62.71

PQI 14 - Uncontrolled Diabetes Admission Rate 93.95 93.28 96.83 93.22

PQI 15 - Asthma in Younger Adult Admission Rate 99.01 96.41 98.53 100.00

PQI 16 - Lower Extremity AMP Among Patients with Diabetes Rate 99.11 98.74 99.12 100.00

PQI 90 - Overall PQI Composite 97.22 89.23 93.60 100.00

Color legend >90 80-89 70-79 <70

2014 MHS Review Group

Source: Military Health System Mart (M2), July 2014

Review of available data demonstrated no meaningful difference overall between CONUS and 

OCONUS PQI values for any of the Services.

 Recommendation Regarding MHS Performance Against PQI Measures

a. MHS governance should implement provider level PQI education followed by an 

evaluation of MTF utilization of AHRQ PQI measures and implementation of a 

monitoring program requiring improvement plans as indicated.

99
August 29, 2014 4. Quality of Care in the Military Health System

Thirty-Day Readmissions 

Hospital readmissions within 30 days may occur due to an unrelated diagnosis or a planned 

course of treatment. An increased focus on readmissions has occurred across the health care 

industry due to the perception that some are the result of poor care or a lack of coordinated care 

and thus may be avoidable. A MTF initiative to study and improve 30-day readmissions is an 

example of successful collaboration within the DoD and with external partners through the 

framework of Partnership for Patients (PfP). 

The direct care component established a plan to implement readmissions prevention strategies in 

2011. Given the lack of a nationally recognized standard methodology in the measurement of 

readmission, the Air Force Health Informatics Division developed a research-based methodology 

to assess MTF readmission rates. The methodology used observed rates (without riskadjustment), and followed several prominent research studies, which excluded cancer patients, 

obstetric and perinatal patients, rehabilitation patients, as well as patient transfers. 

Expanding on the efforts from PfP, DHA recently initiated a registry identifying inpatients at 

high risk for readmission. Within 24 hours of admission, MTF staff can view the names of 

patients in high readmission risk categories in the Population Health Portal. Post-discharge 

follow-up appointment tracking is included on the site as well as data on the facilities’ top 

readmission diagnoses and rates. 

MHS-Level Discussion: MTFs performed well over the past four years, achieving a 10-percent 

reduction in rate of readmissions overall (from 9.8 to 8.8 percent). However, continued 

improvement is required to meet the PfP goal of a 20-percent reduction in readmissions. There 

are no national benchmarks available with this research-based methodology; however, 

standardized readmission measures have since been established and are now available. DHA is 

transitioning to the NCQA HEDIS® all-cause readmissions measure. 

Service-Level Discussion: From 2010 to 2013 results indicate that each Service showed 

improvement in reducing readmissions. The Army and Air Force had the greatest reduction in 

readmissions, but also had the highest readmission rates in 2010. The Navy improved slightly, 

having an initial overall readmission rate of 8.45 percent, the lowest among the Services. 

Though the NCR MD inpatient facilities were not completely realigned until 2012, their 

readmissions rate decreased by 6 percent over the same period (Table 4.8). 

Table 4.8 Readmission Rates According to Services, 2010 – 2013

Branch 2010 2011 2012 2013 Percent Change

(2010 to 2013)

AIR FORCE 10.77 10.22 9.30 9.40 - 12.73

ARMY 10.04 9.44 8.54 8.74 - 12.96

NAVY 8.45 8.61 8.25 8.37 - 0.92

NCR 10.56 11.81 9.58 9.85 - 6.73

2014 MHS Review Group

Source: MHS Population Health Portal, June 2014

100
Military Health System Review – Final Report August 29, 2014

Facility Type, Location, and Other Discussions: Both military medical centers (MEDCEN) 

and smaller hospitals achieved reduced readmissions from 2010 to 2013 (Table 4.9). The 

variance in readmission rates is not directly attributed to size as MEDCENs and hospitals are 

found on both ends of the spectrum (better than expected and worse than expected). (See 

Appendix Figure 4.4-3 for facility-specific rates).

Table 4.9 Readmission Rate According to Facility Type, 2010 – 2013

Facility Type 2010 2011 2012 2013 Percent 

Difference Percent Change

MEDCEN 10.93 10.56 9.70 9.83 -1.10 -10.10

HOSPITAL 7.48 7.08 6.11 6.42 -1.06 -14.17

2014 MHS Review Group

Source: MHS Population Health Portal, June 2014

Location: Readmission rates for OCONUS facilities were similar to the overall rate of smaller 

hospitals, since the majority of overseas facilities are classified as small hospitals. 

Comparison to External Health Systems: Hospital Compare data indicate the U.S. national 

unplanned readmission rate is 16 percent. Data were provided by two health systems for 

comparison. The readmissions methodology used by both was all cause 30-day readmission rate. 

As the methodologies used to calculate the data differ for the health systems, Hospital Compare,

and the MTFs, the readmission rates are not comparable. However, reviewing the percent 

change in the readmission rates indicates the MTFs are doing well in reducing readmission rates 

(Table 4.10). 

Table 4.10 Readmission Rate Comparison to External Health Systems, 2010 – 2013

Health Plan 2010 2011 2012 2013 Percent Change

MHS (Unadjusted) 9.80 9.40 8.60 8.80 -10.20

HS2 (Average) 10.54 10.85 11.18 11.06 1.04

HS3 (Non-Medicare) 6.73 6.33 6.31 6.22 -7.60

HS3 (Medicare) 11.52 11.91 11.80 11.61 1.00

2014 MHS Review Group

Source: MHS Population Health Portal, June 2014

In summary, all Services demonstrated reduced 30-day all-cause readmission rates as part of a 

coordinated, collaborative effort through the PfP initiative. However, the goal of a 20-percent 

reduction was not met.

 Recommendations Regarding Readmission Rates

a. MHS governance should establish an implementation plan for the MHS Population 

Health Portal readmissions site to ensure maximum utilization so as to reduce 

avoidable readmissions.

101
August 29, 2014 4. Quality of Care in the Military Health System

b. The DHA Healthcare Operations Directorate should complete transition to the 

HEDIS® All-Cause Readmission standardized measure, which is risk-adjusted and 

has national benchmarks.

National Perinatal Information Center (2010-2013) 

Within the MHS, 52 MTFs provide inpatient obstetrical care and deliver approximately 50,000 

infants annually. The purchased care obstetrical facilities deliver an additional 50,000 infants; 

however, this section describes measures related to direct care only. 

The National Perinatal Information Center (NPIC) provides the MHS with quarterly direct care 

data, presented as comparisons to averages of civilian hospitals participating as members in the 

NPIC/Quality Analytic System proprietary Perinatal Center Data Base (PCDB). NPIC averages 

are based on 86 civilian facilities. Data points are provided for 10 descriptive measures and 10 

comparative measures. The comparative measures have a comparable NPIC average and 

provide a reasonable and accepted assessment of quality of obstetric care (Table 4.11).

Table 4.11 National Perinatal Information Center – Executive Summary Measures

2014 MHS Review Group

Source: National Perinatal Information Center, July 2014

MHS-Level Discussion 

Descriptive Measures: During the past four years the number of deliveries in the direct care 

component has remained consistent at 50,000 annual deliveries. Army (21 facilities) delivers 52 

percent of infants, Navy (17 facilities) delivers 31 percent, Air Force (12 facilities) delivers 12 

percent, and the NCR MD (2 facilities) delivers 5 percent of infants (Appendix Tables 4.4-14a 

and Appendix Table 4.4-14b). For the past four years, the percent of Cesarean section (CDescriptive Measures Comparative Measures

Maternal Maternal

1) Total number of deliveries

2) Percent of induction of labor

3) Percent of Cesarean Section 

4) Percent of operative deliveries – broken down 

to percent of forceps and vacuum

5) Maternal readmission to other than delivery site

1) Percent of inductions less than 37 weeks of 

pregnancy with medical indications

2) Percent of Cesarean Section less than 37 

weeks of pregnancy with medical indications

3) Patient safety indicator 18 (OB trauma with 

instruments)

4) Patient safety indicator 19 (OB trauma without 

instruments)

5) Vaginal delivery with shoulder dystocia

6) Postpartum hemorrhage

7) Maternal readmission to delivery site

Neonatal Neonatal

1) Total number of neonate

2) Percent of neonates born in hospital (inborn)

3) Percent inborn less than 1500 grams

4) Percent of neonates with non-routine bed days

5) Inborn readmission to any facility

1) Inborn readmission to birth facility

2) Patient safety indicator 17 (injury to neonate)

3) Inborn mortality greater than or equal to 500 

grams

102
Military Health System Review – Final Report August 29, 2014

section) deliveries in the MTFs (across all Services) has remained lower than the NPIC average 

(26 percent in the MTFs in 2013 as compared to 35 percent for NPIC average) and the rate of 

obstetrical forceps deliveries in the MTFs has been higher than the NPIC average for the past 

four years (Appendix Table 4.4-14c). The MHS average of forceps deliveries has been 

somewhat stable over the past four years (1.5 percent, 1.8 percent, 1.5 percent, and 1.5 percent) 

while the NPIC average has decreased each year over the past four years from a high of 1.1 

percent in 2010 to a low of 0.9 percent in 2013.

While there is no optimal benchmark for operative vaginal delivery rates, the finding that MTF 

providers perform a higher number of forceps vaginal deliveries is in accord with the American 

Congress of Obstetricians and Gynecologists (ACOG) position that supports the use of forceps 

and vacuum deliveries when safe and indicated to help reduce rates of C-section deliveries.48 

Comparative Measures

Percent of Inductions and C-section Deliveries Prior to 37 Weeks Gestation with a Medical 

Indication: Delivery of infants prior to 37 weeks of pregnancy is considered preterm delivery. 

The direct care component outperformed the NPIC average for each of the past four years in the 

appropriate delivery of preterm infants (Appendix Figures 4.4-4a and 4.4b). High rates in these 

two measures (over the NPIC averages) indicate that in the vast majority of cases, the direct care 

component performs induction of labor or C-sections in preterm mothers only when medically 

indicated. 

Patient Safety Indicators (PSIs) 18 and 19: AHRQ defines PSIs to track potential harm 

events. PSI 18 and 19 are measures of injury to the mother during vaginal delivery. The direct 

care rate for PSI 18 (injury during delivery with instruments) has been lower than the NPIC 

average from 2010 to 2013, with the exception of 2012, when it was higher (Appendix Figure 

4.4-5). The direct care rate for PSI 19 (injury during deliveries without instruments) was lower 

than the NPIC average each year from 2010 to 2013 (Appendix Figure 4.4-6). The PfP initiative 

to reduce harm addressed the need to decrease harm events for PSI 18 and 19, including SAFER 

PASSAGES, a mnemonic tool developed by the Air Force to assist delivering providers. A 22-

percent reduction in the rate of PSI 18 and an 8 percent reduction in PSI 19 were observed during 

the implementation of this initiative (2012 to 2013) from the 2010 baseline.

Shoulder Dystocia: Shoulder dystocia is neither predictable nor preventable and risk factors 

(e.g., obesity and gestational diabetes) are increasing in the population as a whole. The direct 

care and NPIC rates of shoulder dystocia have each increased 0.25 percent from 2010 to 2013 

and the direct care rate has remained above the NPIC average since 2010 (Figure 4.2). 

48Safety Prevention of Primary Cesarean Delivery. American College of Obstetrics and Gynecology. Available at: 

http://www.acog.org/Resources-And-Publications/Obstetric-Care-Consensus-Series/Safe-Prevention-of-thePrimary-Cesarean-Delivery. Accessed: July 25, 2014.

103

 
August 29, 2014 4. Quality of Care in the Military Health System

Figure 4.2 Annual Rate of Vaginal Deliveries Coded with Shoulder Dystocia, CY10 – CY13

2014 MHS Review Group

Source: National Perinatal Information Center/Quality Analytic Services (NPIC/QAS), July 2014.

Postpartum Hemorrhage (PPH): The direct care component averages of PPH have increased 

and have remained significantly higher over the past four years as compared to the NPIC 

average, which has remained flat since 2010 (Figure 4.3). 

Figure 4.3 Annual Postpartum Hemorrhage Rate, CY10 – CY13

2014 MHS Review Group

Source: National Perinatal Information Center/Quality Analytic Services (NPIC/QAS), July 2014.

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

2010 2011 2012 2013

Rate

CY

MHS Facilities Average Overall NPIC Database Average

Linear (MHS Facilities Average) Linear (Overall NPIC Database Average)

0%

1%

2%

3%

4%

5%

6%

2010 2011 2012 2013

Rate

CY

MHS Average Overall NPIC Database Average

Linear (MHS Average) Linear (Overall NPIC Database Average)

104
Military Health System Review – Final Report August 29, 2014

Review of the direct care perinatal program revealed that existing education and training 

capabilities that address shoulder dystocia and PPH have been implemented but have not yielded 

the desired level of improvement. The Mobile Obstetric Emergencies Simulator (MOES) is a 

comprehensive in-situ obstetric emergency simulation platform. The Association of Women’s 

Health, Obstetrics and Neonatal Nursing (AWHONN) and the American Academy of Family 

Physicians (AAFP) have training programs that support collaborative care and have been used as 

foundational training for obstetrical staff.

Postpartum and Infant Readmission to Delivery Site: These measures examine the rate of 

postpartum readmission to MTF within 42 days post discharge or infant readmission to the 

delivery site at less than 29 days of age (Appendix Figure 4.4-7 and Appendix Figure 4.4-8). 

The direct care average has been higher than the NPIC average for both measures in the past 

three years; maternal readmission was lower for the direct care component than NPIC in 2010. 

AHRQ PSI 17: PSI 17 measures birth trauma, injury to infant, per 1,000 newborns, excluding 

certain categories of infants with specific conditions.49 This metric is based on coding of one of 

six specific conditions or the category of “other specified birth injury.” Direct care average 

annual rate of PSI 17 injury to neonates has remained higher than the NPIC average from 2010 

to 2013 (Figure 4.4). A closer look at data from 2013 illustrates a consistent issue with this 

metric (Figure 4.5). The overall direct care rate for PSI 17 in 2013 was 0.4 percent compared to 

0.2 percent for the NPIC average, but the category of “other specified birth injury” accounted for 

more than 65 percent of the 227 direct care PSI 17 cases. The large percentage of injury that 

cannot be categorized needs further review.

49 PSI 17 excluded preterm infants with a birth weight less than 2,000 grams, infants with any diagnosis code of 

injury to brachial plexus, and infants with any diagnosis code of osteogenesis imperfecta.

105

 
August 29, 2014 4. Quality of Care in the Military Health System

Figure 4.4 Annual Rate of PSI 17 Injury to Neonate, CY10 – CY13

2014 MHS Review Group

Source: National Perinatal Information Center/Quality Analytic Services (NPIC/QAS), July 2014.

Figure 4.5 2013 Direct Care Breakout of Inborn Birth Trauma Occurrences by Diagnostic Code

Note: Number of inborns was 49,217 in 2013: 227 is the total number of affected inborns coded within PSI-17.

2014 MHS Review Group

Source: National Perinatal Information Center/Quality Analytic Services (NPIC/QAS), July 2014

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

2010 2011 2012 2013

Rate

CY

MHS Average Overall NPIC Database Average

Linear (MHS Average) Linear (Overall NPIC Database Average)

Subdural or 

cerebral 

hemorrhage, 19, 

(8%)

Epicranial 

subaponeurotic 

hemorrhage, 30, 

(13%)

Other injuries to 

skeleton, 10,( 5%)

Injury to spine and 

spinal cord , 0, (0%)

Facial nerve injury, 

18,

( 8%)

Other cranial and 

peripheral nerve 

injuries, 2, (1%)

Other specified 

birth trauma, 148, 

(65%)

.

106
Military Health System Review – Final Report August 29, 2014

Infant Mortality: The infant mortality measure includes inborn, in-hospital deaths per 1,000 

neonates, weighing at least 500 grams at birth and excluding newborns with specific congenital 

conditions. The direct care average for infant mortality has been lower than NPIC from 2010 to 

2013. The NPIC average has been decreasing over the past four years (from 3.5 to 2.5 percent), 

while the direct care average has been relatively flat (at 1.5 percent all four years) (Figure 4.6). 

The lower direct care rate may reflect appropriate decision making on the part of the providers to 

transfer preterm and high-risk patients to more appropriate settings for delivery as indicated. 

Figure 4.6 Infant Mortality Rate (per 1,000 live births) for Infants Weighing 500 Grams or Greater, 

CY10 – CY13

2014 MHS Review Group

Source: National Perinatal Information Center/Quality Analytic Services (NPIC/QAS), July 2014.

Service-Level Discussion: For metrics in which the direct care component has been at or better 

than the NPIC average (prevention of delivery of preterm infants without medical indications, 

PSI 18 and 19, infant mortality), Service-level data has been consistent with overall direct care 

data (Table 4.12).

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

2010 2011 2012 2013

Rate

CY

MHS Average Overall NPIC Database Average

Linear (MHS Average) Linear (Overall NPIC Database Average)

107
August 29, 2014 4. Quality of Care in the Military Health System

Table 4.12 Perinatal Performance Measures Summary – 2013 Annual Rates

No. Measure NPIC 

Average

Direct 

Care 

Average

Army Air Force Navy NCRMD

1 Percent of inductions prior to 37 weeks 

gestation with a medical indication 97.1% 99.2% 98.3% 100.0% 99.7% 100.0%

2 Percent of C-sections prior to 37 weeks 

gestation with a medical indication 94.8% 96.0% 92.7% 98.9% 98.4% 100.0%

3 PSI 18 - Obstetric Trauma-Vaginal 

delivery with instruments 0.1481 0.1449 0.1285 0.1145 0.1604 0.2703

4 PSI 19 Obstetric Trauma-Vaginal 

delivery without instruments 0.0207 0.0177 0.0178 0.0167 0.0165 0.0262

5 Shoulder dystocia 0.0226 0.0258 0.0277 0.0369 0.0279 0.0202

6 Postpartum hemorrhage (PPH) 0.0340 0.0507 0.0464 0.0570 0.0562 0.0461

7 Postpartum Readmission to Delivery 

Site 0.0098 0.0136 0.0149 0.0110 0.0125 0.0139

8 Inborn Readmission to Delivery Site 0.0113 0.0342 0.0343 0.0301 0.0387 0.0152

9 PSI 17 Birth Trauma-Injury to Neonate 0.0020 0.0043 0.0041 0.0047 0.0042 0.0072

10 Inborn Mortality Rate >= 500 Grams 0.0026 0.0016 0.0014 0.0003 0.0024 0.0012

 Rates are statistically significantly better than the NPIC average

 Rates are statistically significantly worse than the NPIC average.

 Rates are observationally better than the NPIC average; but there was not enough data 

for calculation of confidence intervals thus statistical significance could not be determined.

Lower is better for all measures except Measures 1 and 2.

2014 MHS Review Group

Source: National Perinatal Information Center Database, July 2014

For PSI 18 and 19, the average rate for all Services has been within the confidence interval or 

below the NPIC average. The Air Force demonstrated the greatest decrease in rates of PSI 18 or 

19 during the period of the PfP initiative when the program SAFER PASSAGES was 

implemented. 

Service-level data for shoulder dystocia reveal that Air Force and Navy have been two standard 

deviations above the NPIC average from 2011 to 2013, while the Army has been within 

confidence intervals for the past four years (Appendix Figure 4.4-9). Consistent with direct care 

data, Service and NCR MD averages for PPH and for PSI 17 are significantly higher than the 

NPIC average from 2010 to 2013 (Appendix Figures 4.4-10 and 4.4-11). From 2011 to 2013, 

postpartum and neonatal readmission rates for all three Services have been above the NPIC 

average. In 2010, there was between-Service variability for postpartum readmission, but direct 

108 
Military Health System Review – Final Report August 29, 2014

care remained above the NPIC average. Direct care and Service neonatal readmission rates were 

elevated for all four years. 

NCR MD data demonstrate statistically significantly higher averages than the NPIC averages in 

PSI 18, PPH, and PSI 17 (Table 4.12). Administrative issues with coding identified during NPIC 

data collection at the two NCR MD institutions raise concerns with the validity of NCR MD 

data.

Facility Type, Location, and Other Discussions: MTFs that were two standard deviations 

(clinically significant interval) above the NPIC average for rate of shoulder dystocia, PPH, and 

PSI 17 for two or more out of four consecutive years (“consistent outlier”) were identified 

(Appendix Table 4.4-15). A total of 11 MTFs (4 Army, 3 Navy, 4 Air Force) were identified to 

be consistent negative outliers for shoulder dystocia. A total of 25 MTFs (8 Army, 8 Navy, 8 Air 

Force, and 1 NCR MD) were identified to be consistent negative outliers for PPH. A total of 7 

MTFs (3 Army, 2 Navy, 1 Air Force, and 1 NCR MD) were found to be consistent negative 

outliers for PSI 17. 

Comparison to Two External Health Systems: Data were available to compare the 

performance of the direct care component with two external health systems (HS2 and HS3) for 

PSI 17, 18, and 19. The direct care component is underperforming compared to the two health 

systems in PSI 17. As discussed above, “other specified birth injuries” makes up the majority of 

codes for PSI 17 cases and comparative data about the breakdown of code categories are not 

available from the NPIC database or the external health systems. Direct care review is needed to 

determine if these findings are attributable to quality-of-care issues, over-coding, or a 

combination of both. 

While PSI 18 has decreased (improved) in the last three years, and MTFs are performing at the 

NPIC average, the direct care component is underperforming in comparison with HS2 and HS3 

in this metric. For PSI 19, the direct care component is performing well, with rates better than 

the NPIC average and HS3. However, HS2 is outperforming both direct care and the NPIC 

average in PSI 18 and PSI 19. An important caveat is that HS2 serves a demographically 

different population and it is unknown to what extent that may affect these rates.

Summary of Findings:

1. The direct care component met or outperformed the NPIC average in 5 of 10 comparative 

measures (percent of inductions less than 37 weeks with medical indication, percent of Csection less than 37 weeks with medical indication, PSI 18, PSI 19, and infant mortality).

2. The direct care component underperformed compared to NPIC average in 5 of 10 

comparative measures (shoulder dystocia, PPH, PSI 17, maternal readmission to delivery 

site, and inborn readmission to birth facility). There is evidence that data integrity and 

coding issues were potentially responsible for a component of the identified trends. A 

109
August 29, 2014 4. Quality of Care in the Military Health System

2005 Patient Safety Study found that less than 22 percent of charts were coded accurately 

for infant injury.50 

3. The Tri-Service Perinatal Advisory Group (PAG) has made significant strides in internal 

and external collaborations to address important quality issues in obstetric care, including 

PfP, MOES, and use of AWHONN and AAFP training and education programs to 

improve the quality of perinatal care. In some cases, these initiatives have been 

associated with improvement in outcomes, but improvement has not been demonstrated 

in all measures (shoulder dystocia, PPH, and, and PSI 17). There appears to be 

inconsistent implementation of available programs and tools. 

 Recommendations Regarding MHS Perinatal Services

a. MHS governance should require a review of perinatal provider documentation and 

coding practices at MTFs to validate data integrity.

b. MHS governance should ensure that standardization of accurate perinatal coding 

practices is implemented across direct care.

c. MHS governance should investigate readmissions of mothers and infants. This 

clinical review of diagnostic codes at readmission will identify the medical 

conditions that drive these rates and determine if lagging performance is a quality 

issue or related to military-unique issues and flexibility.

d. Health Affairs policy is needed to standardize annual and interval training 

requirements related to perinatal care.

e. The Perinatal Advisory Group should conduct a comprehensive review of clinical 

practices related to metrics where MHS is underperforming. Through a dashboard 

and standardized metric reporting requirements, intervention plans should be 

developed and actions prioritized. 

National Surgical Quality Improvement Program (NSQIP®) 

Analysts reviewed NSQIP® semi-annual reports from each of the 17 participating MTFs from 

July 2010 through June 2013 (last full year of data). NSQIP® reports provided risk-adjusted 30-

day morbidity and mortality outcomes computed for each participating hospital. The NSQIP® 

reported metric is an odds ratio that represents the estimated odds of a complication or event 

occurring in a specific hospital compared to the estimated odds of that event occurring in all 

participating NSQIP® hospitals. A ratio of 1.0 means the hospital was performing as expected. 

A ratio greater than 1.0 means the hospital was doing worse than expected and a ratio less than 

1.0 means the hospital was doing better than expected. All findings noted by NSQIP® and in this 

report were statistically significant with a p-value of 0.05. Three full years of data were not 

available for all 17 hospitals as participation increased over the period and some inpatient 

facilities were closed. Analysts summarized the reports graphically below and identified 

50 Department of Defense. DoD Medical Treatment Facilities Patient Safety Indicator 17, Birth Trauma. August 

2005. 

110

 
Military Health System Review – Final Report August 29, 2014

statistically significant outliers. Analysts described facilities performing statistically much better 

than expected in any category as “exemplary,” an outcome similar to other facilities included in 

the model as “as expected,” and those performing statistically much worse than expected as 

“needs improvement.” 

Limitations are as follows:

1) Only 17 of 56 MTFs participate in NSQIP®; consequently, these findings may not be 

representative of all MTFs.

2) Until recently, NSQIP® has excluded low-volume facilities from participation; therefore, 

the NSQIP® sample is overrepresented by larger, high-volume facilities. For this reason, 

the NSQIP® civilian sample is unlikely to be representative of all U.S. hospitals. It is 

unclear if the standards presented by NSQIP® are generalizable to all U.S. hospital 

surgical programs or to the direct care component as a whole.

3) While civilian institutions participate in NSQIP®, there are no specific TRICARE 

purchased care data available in the NSQIP® data set. Thus, the NSQIP® sample 

represents direct care only.

MHS-Level Discussion: ALL CASE mortality, displayed in Figure 4.7, and in all subsequent 

tables, which summarizes all surgical care at a facility, was as expected for all 17 participating 

MTFs over the entire study period. A single exception occurred in the June 30, 2012 report for 

one facility where mortality returned to “as expected” in the next reporting period. The majority 

of facilities demonstrated “as expected” or “exemplary” morbidity rates during the reporting 

period. In the most recent reporting year (July 2012 to June 2013), eight facilities demonstrated 

higher-than-expected levels of morbidity and were identified as “needing improvement.”

111
August 29, 2014 4. Quality of Care in the Military Health System

Figure 4.7 ALL-CASE Mortality and Morbidity, MTFs, 2010 – 2013

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

As depicted in Figure 4.8, the major contributors to this morbidity were Urinary Tract Infection, 

Surgical Site Infection, and Return to the Operating Room. Mortality Morbidity Mortality

MorbidityMortality Morbidity

David Grant Med Ctr #109 - * - * - *

Michael O'Callaghan Federal Hospital #2057 - - - 

Blanchfield Army Med Ctr #283 -  - - - -

Carl R Darnall Army Med Ctr #504 - - - * - -

Eisenhower Army Medical Center # 284 - * - * * *

Madigan Army Med Ctr #282 -  -  - 

San Antonio Military Med Ctr #294 -  -  - 

Tripler Army Med Ctr #287 -  - * * *

William Beaumount Army Med. Ctr #125 - - - - - *

Womack Army Med Ctr #281 - -   - 

Naval Med Ctr San Diego #503 -  - - - 

Naval Med Ctr Portsmouth #501 -  -  - 

NCR Walter Reed Nat'l Mil Med Ctr #500 - - -  - *

ARMY Evans Army Comm Hosp #505 - - - - - 

Naval Hospital Jacksonville #502 - * - - - -

Naval Hospital Pensacola - -

NCR Fort Belvoir Community Hospital #2062 - - - 

* Exceeds Standards - Meets Standards

 Needs Improvement Data Unavailable

July '12-

June '13

NAVY

NAVY

Medical Center Community Hospitals

July '11-

June '12

July '10-

June '11

AIR 

FORCE

ARMY

112
Military Health System Review – Final Report August 29, 2014

Figure 4.8 Morbidity Trends across MTFs, 2010 – 2013

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

*

0 3 6 9 12 15

 Cardiac

 Unplanned Intubation

 Ventilator>48 Hours

 Pneumonia

 DVT/PE

 Renal Failure

 SSI

 UTI

ROR

 Cardiac

 Pneumonia

 Unplanned Intubation

 Ventilator>48 Hours

 DVT/PE

 Renal Failure

 UTI

 SSI

 ROR

 Cardiac

 Pneumonia

 Unplanned Intubation

 Ventilator>48 Hours

 DVT/PE

 Renal Failure

 UTI

 SSI

 ROR

June '10-July '11 June '11-July '12 June '12-July '13

Number of MTFs

Needs Improvement

Meets Standards

* Exceeds Standards

113
August 29, 2014 4. Quality of Care in the Military Health System

The summary of morbidity outcomes for all facilities through the entire study period is presented 

in Figure 4.9.

Figure 4.9 MTF Comparison by Type of Morbidity Outcome, 2010 – 2013

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

Authors of this report discussed MTF performance with NSQIP® officials in an attempt to assess 

direct care systemwide performance against other systems. The officials reported that many 

large hospital systems have varying levels of performance among their facilities. It was also 

noted that as facilities within hospital systems begin to collaborate those variances in 

performance often improve.

Service-Level Discussion: The Army has nine facilities participating, the Navy four facilities, 

the Air Force two facilities, and the NCR two facilities (Figure 4.10). Cardiac Pneumonia Unplanned Intubation Ventilator>48 Hours DVT/PE Renal Failure UTI SSI ROR Cardiac Pneumonia Unplanned Intubation Ventilator>48 Hours DVT/PE Renal Failure

 UTI SSI ROR Cardiac Pneumonia Unplanned Intubation Ventilator>48 Hours DVT/PE Renal Failure UTI SSI ROR

David Grant Med Ctr #109 - - * * - - - - - * * * - - * - - - - - - - - - - -

Michael O'Callaghan Federal Hospital #2057 - - - - - - - -  - - - - - - - - -

Blanchfield Army Med Ctr #283 - - - - - - - - - - - - - - - - - - - - - - - - - -

Carl R Darnall Army Med Ctr #504 - - - - - - - - - - - - - - - - - - - - - - - - - -

Eisenhower Army Medical Center # 284 - - - - * - - * - * - - * * - * - - * * - - * - * -

Madigan Army Med Ctr #282 -  - - -   -  - - - - -   -  - - - - -   

San Antonio Military Med Ctr #294 - - - - - -   -  - - - -   - - - - - - -  - 

Tripler Army Med Ctr #287 - - - - - -   - - - - - - - * - - * - * - * - - *

William Beaumount Army Med. Ctr #125 - - - - - - - - - - -  - - - - - - - -  - - * * -

Womack Army Med Ctr #281 - - - - * - - - - - -  - - - -  - - - - - - - - -

Naval Med Ctr Portsmouth #501 - - - - - -  - - - - - - - -  - - - - - - - -  -

Naval Med Ctr San Diego #503 -  - -  - -  * - - - - - - - - - - -  - -  - -

NCR Walter Reed Nat'l Mil Med Ctr #500 - * - -   - - - -  -  -   - - - - - - * - * 

ARMY Evans Army Comm Hosp #505 - - - - - - - - - - - - - - - - - - - - - - -  - -

Naval Hospital Jacksonville #502 - - - - - - * * - - - - - - - *  - - - - - - - - 

Naval Hospital Pensacola - - - - - - - - -

NCR Fort Belvoir Community Hospital #2062 - - - - - - - - - - - - - - - -  -

* Exceeds Standards - Meets Standards  Needs Improvement Data Unavailable

July '10 - June '11 July '11 - June '12 July '12 - June '13

Medical Centers Community Hospitals

AIR FORCE

ARMY

NAVY

NAVY

114
Military Health System Review – Final Report August 29, 2014

Figure 4.10 Service-Level Comparison, 2010 – 2013

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

Figure 4.11 Number of MTFs that Need Improvement, Meet Standards, or Exceed Standards in 

Post-surgical Morbidity by Facility Type (2010 – 2013)

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

*

* *

* * *

*

*

0

1

2

3

4

5

6

7

8

9

10

July '10 - June '11July '11 - June '12

July '12 - June '13July '10 - June '11

July '11 - June '12July '12 - June '13July '10 - June '11July '11 - June '12July '12 - June '13

July '10 - June '11July '11 - June '12

July '12 - June '13

Air Force Army Navy NCR

Number of MTFs

Needs Improvement Meets Standards * Exceeds Standards

*

* * *

0

2

4

6

8

10

12

14

July '10 - June '11July '11 - June '12July '12 - June '13July '10 - June '11July '11 - June '12July '12 - June '13

Community Hospitals Medical Centers

Number of MTFs

Needs Improvement Meets Standards * Exceeds Standards

115
August 29, 2014 4. Quality of Care in the Military Health System

Facility Type: As shown in Figure 4.11, medical centers make up 11 of the 17 participating 

MTFs (65 percent); thus, they have the highest number of facilities requiring improvement, but 

also contain the largest number of exemplary performers. 

Location and Purchased Care: There are no OCONUS hospitals currently participating in 

NSQIP®. Furthermore, a limitation of this inquiry is that there are no specific purchased care 

data from NSQIP® by which to compare MTF surgical outcome data. 

Comparison to External Health Systems: Comparative NSQIP® data were received from 

Health System 3 (HS3). The data provided were number of morbidity events per 1,000 

admissions in each NSQIP® category. The data provided in the NSQIP® semi-annual report 

include the observed surgical event rates as well as the risk-adjusted expected rates, which are 

based on the patient comorbidities and the complexity of the procedure. For comparison, MTF 

data, both morbidity and mortality, were converted to the number of events per 1,000 admission 

and are displayed in Figure 4.12. 

Although not risk adjusted, there is a general downward trend in this rate in all categories over 

the period of the study.

Figure 4.12 MHS Surgical Mortality and Morbidity Rates July 2010 – June 2013

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

0

10

20

30

40

50

60

Events per

1000 cases

July '10-June '11 July '11-June '12 July '12-June '13

116
Military Health System Review – Final Report August 29, 2014

Figure 4.13 MHS and HS3 Surgical Morbidity Rates, July 2012 – June 2013

2014 MHS Review Group

Source: American College of Surgeons National Surgical Quality Improvement Program, Semiannual Reports, July 

2014

When comparing surgical outcomes, using risk-adjusted rates is the preferred method. The MTF 

data are compared to HS3 in Figure 4.13. The comparability for this data is limited as it is not 

risk adjusted; thus, it does not take into account pre-existing patient conditions that could affect 

the surgical outcome. Results indicate that HS3 has higher numbers of surgical morbidity events 

across the spectrum as compared with direct care MTFs.

Direct care data obtained from participation in PfP showed a 36.5 percent reduction (Figure 4.14) 

which is similar to the 42 percent reduction seen in NSQIP® direct care rate seen in Figure 4.12.

0

2

4

6

8

10

12

14

16

18

Events per 

1000 cases

MHS HS3

117
August 29, 2014 4. Quality of Care in the Military Health System

Figure 4.14 Direct Care Partnership for Patients (PfP) Surgical Site Infection Rates, 2010 – 2013

2014 MHS Review Group

Source: Direct care Partnership for Patients, July 2014

Summary of Findings:

1. There is evidence in direct care of a long-term commitment to collect surgical outcome 

data and improve performance of facilities: 30 percent of eligible direct care MTFs 

participate in NSQIP® compared to 10 percent of U.S. civilian hospitals.

2. Three MTFs in the most recent data period are performing at the top tier nationally.

3. Surgical morbidity is statistically significantly higher than expected in eight MTFs and 

was sustained over the reporting period in several of these facilities.

4. Morbidity involving Urinary Tract Infection, Surgical Site Infection, Return to the 

Operating Room, and Pneumonia has been most problematic for facilities.

5. Surgical mortality in the most recent data period is as expected in all 17 facilities.

 Recommendations Regarding Surgical Quality Improvement

a. MHS governance should explore expanding NSQIP® participation to all remaining 

direct care inpatient facilities performing surgery. In addition, it should ensure all

ambulatory surgery platforms participate in a similar surgical quality improvement 

program.

b. The DHA Healthcare Operations Directorate should partner with the American 

College of Surgeons NSQIP staff to improve MTF collaboration and sharing of best 

practices of top performing facilities, thereby decreasing overall direct care surgical 

morbidity and improving clinical outcomes. 

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

2010 2011 2012 2013

SSI Rate

FY

%SSI

36.5 % 

Reduction

118
Military Health System Review – Final Report August 29, 2014

c. MHS governance should task the NSQIP® working group to assess surgical 

morbidity shortfalls to the Medical Operations Group for Tri-Service/DHA 

engagement, collaborative support, and facility action.

Inpatient Mortality Measures

While inpatient mortality has traditionally not been viewed as an accurate reflection of care 

quality, a consensus among leading civilian organizations is that the judicious use of riskadjusted mortality measures can serve a valuable role in identifying trends warranting further 

investigation. Risk-adjusted, disease-specific and condition/procedure-specific mortality rates 

are more accurate and useful in quality management.

AHRQ developed condition-specific mortality measures using administrative records as part of 

its Inpatient Quality Indicator (IQI) measure set. The IQI measure set contains a number of 

condition-specific mortality measures including deaths from Pneumonia, Acute Myocardial 

Infarction (AMI), Stroke, and Congestive Heart Failure. Data are risk-adjusted to account for the 

different risk of death among patient populations and can be used to identify higher-thanexpected condition-specific mortality rates at a given facility compared to other facilities. 

AHRQ provides benchmark values for its IQI measures and DHA uses these benchmark values 

for comparison. Similar to other risk-adjusted mortality models, the information is used as a 

trigger for additional facility-level investigation. 

IQI mortality measures (Pneumonia, Congestive Heart Failure, AMI, and Stroke) have been 

reviewed by the Clinical Measures Steering Panel but have received limited Service-level action. 

Crude and risk-adjusted mortality measures have only recently been developed at the DHA level 

and have not been used for quality management at the Service or MTF level at this time. No 

measures of mortality for facilities in the purchased care system are reviewed, though facilitylevel data are available online from TJC and Hospital Compare websites. Specific mortality 

measures discussed in this review are displayed in Table 4.13.

Table 4.13 Mortality Measures

Mortality Measures

Acute Myocardial Infarction (AMI) Mortality Rate (IQI #14)

Congestive Heart Failure Mortality Rate (IQI #15)

Acute Stroke Mortality Rate (IQI #16)

Pneumonia Mortality Rate (IQI #20)

Risk Adjusted Mortality Measure by MTF

Source: 2014 MHS Review Group, July 2014

MHS-Level Discussion

IQI Condition Specific Mortality Measures: Numerators for IQI data are small, such that 

single deaths can lead to substantial changes in MTF-level performance from quarter to quarter. 

For this reason, MTFs with less than 20 cases of Stroke, Pneumonia, AMI, or Congestive Heart 

119
August 29, 2014 4. Quality of Care in the Military Health System

Failure per year were excluded from this analysis. Thus, this analysis is largely weighted to 

medical centers for the IQI AMI, Stroke, and Congestive Heart Failure mortality measures but 

includes hospitals (in addition to medical centers) for the IQI Pneumonia mortality measure. 

Table 4.14 illustrates the direct care-level mortality rates from 2010 to 2013 for the IQI AMI, 

Congestive Heart Failure, Stroke, and Pneumonia measures. Numerical values are listed as 

percentages in the table below, along with the IQI average mortality rate in the second column. 

Over the last four years the direct care component has met or exceeded the IQI average for these 

measures 72 percent of the time. IQI 16 – Congestive Heart Failure mortality showed the worst 

performance of all measures with only 62 percent of quarters being within the expected range, 

although all quarters were within 1 to 2 percent of the benchmark value. Values for the other 

measures were 75 percent (Stroke and AMI) and 82 percent (Pneumonia) of quarters within the 

expected range. 

Table 4.14 Direct Care Component IQI Condition-specific Mortality Rates for 2010 – 2013

Below expected mortality

Above expected mortality, warrants investigation. 

*facility with < 10 deaths

2014 MHS Review Group

Source: MHS Population Health Portal, June 2014

Service-Level Discussion

Service-specific quarterly performance has been within the expected range for 69 percent (Army) 

and 72 percent (Navy/Air Force) of this timeframe. Analysis of CONUS and OCONUS sites 

was not completed due to insufficient OCONUS data for comparison.

Assessment of Direct Care Risk Adjusted Mortality: The direct care component has not 

routinely used mortality rates for quality monitoring purposes. For the purposes of this review, a 

study was undertaken to develop risk-adjusted mortality ratios based on a commonly accepted 

(Elixhauser) methodology. Using regression analysis, a statistical model was developed that 

calculated expected deaths based on the case-mix of an MTF’s population for a cohort of large 

hospitals. This initial study only looked at the year 2013, comparing the predicted number of 

deaths for an MTF against the observed number of deaths to define a Standardized Mortality 

Ratio (SMR). A SMR above 1.0 indicated a higher number of deaths than predicted. An SMR 

below 1.0 indicated fewer deaths at the facility than predicted. To ensure these results were 

statistically significant, confidence intervals were calculated such that areas of concern would 

demonstrate an SMR lower confidence limit above 1.0. This model is new and has not been 

IQI 

Measure

Benchmark2010

Q1

2010

Q2

2010

Q3

2010

Q4

2011

Q1

2011

Q2

2011

Q3

2011

Q4

2012

Q1

2012

Q2

2012

Q3

2012

Q4

2013

Q1

2013

Q2

2013

Q3

2013

Q4

CHF 3.2 3.13 2.55 1.87 2.47 3.49 3.28 3.01 4.37 2.08 3.40 3.20 4.01 2.81 1.93 3.32 3.51

Stroke 9.0 10.22 8.53 8.93 7.45 7.75 5.83 7.19 6.72 8.17 5.97 10.52 7.87 10.09 5.06 11.33 6.69

Pneum 3.9 3.36 2.92 1.93 4.15 2.47 3.05 1.72 3.83 2.97 3.00 1.45 4.67 4.16 3.57 2.38 2.75

AMI 6.0 5.62 4.39 4.43 5.63 6.46 6.11 4.36 6.53 4.18 3.19 0.63 4.63 8.34 4.73 2.65 3.46

120
Military Health System Review – Final Report August 29, 2014

presented at the Service or MTF levels; thus there has been no opportunity for MTFs to validate 

these results using accepted methods.

The results of this initial look at risk adjusted mortality in the direct care component for 2013 

showed 16 MTFs with better-than-expected risk adjusted mortality rates, 7 with rates that are 

within the expected range and 4 MTFs with worse-than-expected mortality rates (Table 4.15). 

These results for worse-than-expected mortality rates include facilities that were also identified 

with worse-than-expected IQI mortality rates. All other facilities looked at with this initial 

model are within the expected range for mortality. 

Table 4.15 Facility-specific Risk-adjusted Standardized Mortality Ratios for 2013

Standardized Mortality Ratio (SMR) 

Assessment MTF Discharges

EVANS ACH-FT. CARSON 1910

NH JACKSONVILLE 1623

DARNALL AMC-FT. HOOD 2730

633rd MDG LANGLEY-EUSTIS 913

88th MDG-WRIGHT-PATTERSON 2738

96th MDG-EGLIN 2137

Better than Expected SMR MARTIN ACH-FT. BENNING 2045

99th MDG-O'CALLAGHAN HOSP 2631

FT BELVOIR COMMUNITY HOSP 3221

BLANCHFIELD ACH-FT. CAMPBELL 1273

NH PENSACOLA 1490

WOMACK AMC-FT. BRAGG 4529

WALTER REED NATL MIL MED CTR 8045

SAN ANTONIO MMC 18132

NMC SAN DIEGO 9646

673rd MED GRP-ELMENDORF 1802

EISENHOWER AMC-FT. GORDON 3937

SMR Within Expected Range NH OKINAWA 846

FT. LEONARD WOOD HOSP 1437

MADIGAN AMC-FT. LEWIS 8201

NMC PORTSMOUTH 8651

WILLIAM BEAUMONT AMC 5859

TRIPLER AMC-FT SHAFTER 6992

Worse than Expected SMR NH GUAM-AGANA 1081

60th MED GRP-TRAVIS 4125

81st MED GRP-KEESLER 2342

2014 MHS Review Group

Source: Military Health System Mart (M2), July 2014

Risk Adjusted Mortality: Two external systems provided risk-adjusted mortality data 

calculated using proprietary risk-adjustment formulas. Due to differences in the models used by 

121
August 29, 2014 4. Quality of Care in the Military Health System

direct care component and these external systems, valid comparisons were not possible. These

differences made it impossible to render a valid conclusion regarding differences in the risk 

adjusted mortality rates between systems. 

Summary of Findings:

1. The direct care component lags behind civilian benchmark organizations in the use of 

mortality measures as indicators of potential quality of care problems.

2. Over the last four years the direct care component has met or exceeded the IQI average 

for these measures 72 percent of the time.

3. The results of this risk adjusted mortality model for calendar year 2013 demonstrated 16 

MTFs with better-than-expected risk adjusted mortality rates, 7 with rates that are within 

the expected range, and 4 with worse-than-expected mortality rates. 

 Recommendations Regarding Mortality Measurements

a. MHS governance should integrate measures of mortality into its quality monitoring 

and performance improvement programs.

b. MHS governance should require Service facilities with higher-than-expected 

mortality on an IQI measure for more than one quarter to perform an investigation 

and implement improvement activities as indicated.

c. MHS governance should evaluate the use of the risk-adjusted standardized 

mortality ratio (SMR) model in direct care. Facilities with higher-than-expected 

mortality should validate the risk-adjusted SMR model data and perform a root 

cause analysis as indicated. 

Experience of Care Summary

Results of customer surveys have become increasingly important in measuring MHS 

performance and in directing action to improve beneficiary experience and quality of services 

provided. Surveys are modeled to identify key drivers of satisfaction in the MHS. This 

summary is based on analysis of data from several DoD and Service-specific surveys regarding 

beneficiary experience with MHS health care. The questions chosen for comparison from DoD 

surveys were selected to be comparable to the HCAHPS and CAHPS civilian benchmarks. 

HCSDB is sent quarterly to an annual sample of approximately 200,000 eligible beneficiaries. It 

is emailed to active duty personnel and mailed to other MHS beneficiaries with responses by 

mail or Internet. HSCDB survey data describe the ratings of the patient’s perception of their 

health plan, their health care, their personal physician, and their specialty care. All benchmarks 

for HCSDB are based on the CAHPS 50th percentile. Sampled beneficiaries may or may not 

have used or tried to use healthcare at the time of the survey. Response rates among the surveys, 

and among subpopulations within surveys, vary significantly. 

The TRICARE Outpatient Satisfaction Survey (TROSS) measures outpatient ratings of 

satisfaction with the provider and with health care. TROSS is sent to MHS beneficiaries 

122
Military Health System Review – Final Report August 29, 2014

following outpatient encounters. Approximately 575,000 beneficiaries are surveyed annually. 

All benchmarks for TROSS are based on the CAHPS 75th percentile. 

TRICARE Inpatient Satisfaction Survey (TRISS) measures inpatient discharge patient ratings of 

the hospital overall and whether the patient would recommend the hospital. 

MHS-Level Discussion

According to results from the HCSDB survey, MHS beneficiaries: 

• Are more satisfied with their health plan as compared to the civilian benchmark.

• Rate their overall satisfaction with their health plan at 66 percent in FY 2013, exceeding 

the civilian benchmark of 57 percent. 

• Prime enrollees rate their overall health plan at or above the CAHPS level at 90 percent 

of the MTFs. More than half of the MTFs rate at or above the 75th percentile. 

• Are less satisfied with their health care overall as compared to the civilian benchmark

• Rate their health care at 8, 9, or 10 on a 0–10 scale (64 percent), below the civilian 

benchmark of 72 percent. 

• Showed increased ratings for health care from FY 2011 to FY 2013, according to 

HCSDB direct care ratings. 

• Rate their personal doctor and specialty care below the CAHPS benchmarks. 

• Rate their purchased health care at scores that meet or exceed the CAHPS benchmarks. 

According to TROSS results:

• TROSS scores regarding “satisfaction with the provider” ranked in the middle (50th 

percentile) for most facilities when compared to CAHPS. 

• TROSS scores regarding “overall satisfaction with care” during this time period 

increased from 81 percent in FY 2012 to 84 percent in FY 2013. 

• TROSS scores for “overall satisfaction” with purchased care remained stable between 88 

percent and 89 percent.

According to TRISS results: 

• TRISS ratings for the direct care hospital experience show an increase from 63 percent in 

FY 2011 to 67 percent FY 2013; remaining below the civilian benchmark of 70 percent. 

• TRISS hospital ratings for the Air Force (73 percent) and NCR MD (74 percent) facilities 

were above the benchmark. 

• TRISS scores indicating whether a patient would recommend direct care hospitals 

improved from 68 percent in FY 2012 to 71 percent in FY 2013, reaching the benchmark 

in FY 2013 (71 percent). 

• In FY 2013, the TRISS ratings for “recommend hospital” for the Air Force (76 percent), 

Navy (71 percent), and NCR (81 percent) were all at or above the civilian benchmark; 

while ratings for the Army (68 percent) were below. (The percentiles reported here for 

123
August 29, 2014 4. Quality of Care in the Military Health System

TRISS are not patient mix adjusted; however, they are weighted to remove non-response 

bias and an algorithm is applied to adjust sample weights according to the MHS region 

population distribution).

• Beneficiaries who received care within the purchased care component for surgical and 

obstetric care rated their hospital higher than did those in the direct care component. 

• MHS beneficiaries receiving surgical care in the direct care component rated their 

hospital inpatient experience higher than the civilian benchmark. This remained true in 

FY 2013 for beneficiaries discharged from either a MTF or a civilian hospital. 

• Beneficiaries receiving in-patient obstetric care rated the DoD hospital lower than the 

purchased care hospitals. 

Excluding obstetric data, the overall direct care in-patient scores exceeded the civilian 

benchmark. 

Service-Level Discussion

Service-specific surveys should not be used to compare the different Services health care 

systems. The questions and areas of focus are not comparable.

The Army Provider Level Satisfaction Survey (APLSS), with email and postal mail with web 

response capability, obtains data from approximately 350,000 beneficiaries a year. The survey 

randomly selects patients who may respond to 24 questions by mail or email. The questions are 

designed to gather patient feedback on access to care, cleanliness of the facility, and courtesy of 

the staff. Ratings fluctuated but remained at 93 to 94 percent for overall ratings from FY 2011 to 

FY 2014 QTR 3. In APLSS, specialty care outperforms primary care by four percentage points. 

There is no direct civilian benchmark directly related to the APLSS questions.

The Navy postal and mail response based Patient Satisfaction Survey (PSS) obtains data from 

approximately 134,000 beneficiaries annually. Patient satisfaction results are compared to 

feedback given by the general civilian population regarding their health care providers within 

their private insurance plan. During this time frame, Navy MTFs consistently scored above 90 

percent for overall satisfaction with care. 

The Air Force telephone-based Service Delivery Assessment (SDA) obtains data from 

approximately 200,000 beneficiaries a year. The survey is designed to gather patient feedback in 

multiple areas of concern. SDA quarterly data from FY 2011 Q3 through FY 2014 Q2 indicate 

that ratings for primary care were higher than specialty care. During that timeframe, ratings 

fluctuated but remained over 94 percent. There is no civilian benchmark to compare to this 

measure. 

Facility Type and Location Discussions

TROSS data indicate no large difference in experience of care based on facility type; 

MEDCENs, Community Hospital and Health Clinic) for TROSS. HCSDB ratings for personal 

doctor and health care measures demonstrated small differences. The HCSDB rating of personal 

doctor in the MEDCENs was 74 percent, compared to 69 percent for clinics. HCSDB rating of 

124
Military Health System Review – Final Report August 29, 2014

heath care for medical centers was 59 percent; compared to 56 percent for clinics. APLSS, PSS, 

and SDA scores indicate that within each survey the types of facility score statistically similar 

within a range of two percentage points. 

Location: Scores from HCSDB, TROSS, and TRISS indicate that there are minimal differences 

in the experience of care between CONUS and OCONUS facilities as seen in Table 4.16. It is 

assumed that the differences in the TRISS measure “recommend OCONUS hospital” reflect a 

comparison of the DoD facility with the local national facility. The TRISS CONUS Hospital 

“willingness to recommend” measure matches the civilian benchmark. Service-specific surveys,

APLSS, PSS, and SDA show satisfaction ratings for OCONUS and CONUS to be statistically 

similar in all areas. 

Table 4.16 Ratings of Overseas and U.S. Facilities by OCONUS and CONUS

Measure OCONUS CONUS

HCSDB Ratings of Health Care for Overseas and U.S. facilities 54 % 58 %

HCSDB Ratings of Personal Doctor at overseas and U.S. facilities 69 % 71 %

TROSS Satisfaction with Care ratings for Overseas and U.S. facilities 85 % 83 %

TRISS Hospital Ratings for Overseas and U.S. facilities 66 % 66 %

TRISS Ratings for recommend hospital for overseas and U.S. facilities 82 % 71 %

2014 MHS Review Group

Source: Health Care Survey of DoD Beneficiaries (HCSDB), TRICARE Outpatient Satisfaction Survey (TROSS), 

TRICARE Inpatient Satisfaction Survey (TRISS), July 2014

125
August 29, 2014 4. Quality of Care in the Military Health System

Comparison to External Health Systems

Rating of Health Care Plan: HCSDB TRICARE Prime Enrolled Beneficiaries (MTF and 

Civilian Combined) and the three comparison system are statistically equivalent. During Fiscal 

Years 2010, 2011, and 2012 all four systems, MHS and three external systems (66 to 67 percent),

are significantly higher than CAHPS Benchmark of 57 percent. Statistical significance testing 

was not completed to assess difference in responses (Table 4.17). 

Table 4.17 Rating of Health Care Plan, 2011 – 2013

System 2011 2012 2013

Health System 1 64% 64% 67%

Health System 2 70% 68% 67%

Health System 3 63% 63% 66%

HCSDB TRICARE Prime Enrolled Only (MTF and 

Civilian Combined)

65% 65% 66%

CAHPS Benchmark (adjusted to MHS population) 57% 56% 57%

2014 MHS Review Group

Source: Consumer Assessment of Healthcare Providers and Systems (CAHPS), Health Care Survey of DoD 

Beneficiaries (HCSDB), July 2014

Rating of Personal Doctor: HCSDB TRICARE Prime Enrolled Beneficiaries (MTF and 

Civilian Combined) rated their personal doctor lower than did patients in Health System 2, 

Health System 3, and the CAHPS benchmarks. Statistical significance testing was not 

completed to assess differences (Table 4.18).

Table 4.18 Rating of Personal Doctor, 2011 – 2013 

System 2011 2012 2013

Health System 1 - - -

Health System 2 86% 85% 85%

Health System 3 86% 87% 81%

HCSDB TRICARE Prime Enrolled Only (MTF and Civilian 

Combined)

71% 72% 73%

CAHPS Benchmark (adjusted to MHS population) 80% 79% 80%

2014 MHS Review Group

Source: Consumer Assessment of Healthcare Providers and Systems (CAHPS), Health Care Survey of DoD 

Beneficiaries (HCSDB), July 2014

Health Care Rating: HCSDB TRICARE Prime Enrollees rated their health care similarly to 

enrollees in Health System 1. HCSDB TRICARE Prime Enrollees rated their health care lower 

than did patients in Health System 2, Health System 3, and CAHPS benchmarks. Statistical 

significance testing was not completed to assess differences (Table 4.19). 

126
Military Health System Review – Final Report August 29, 2014

Table 4.19 Health Care Rating, 2011 – 2013

System 2011 2012 2013

Health System 1 58% 59% 62%

Health System 2 81% 79% 79%

Health System 3 77% 77% 79%

HCSDB TRICARE Prime Enrolled Only (MTF and Civilian 

Combined)

59% 61% 62%

CAHPS Benchmark (adjusted to MHS population) 72% 72% 72%

2014 MHS Review Group

Source: Consumer Assessment of Healthcare Providers and Systems (CAHPS), Health Care Survey of DoD 

Beneficiaries (HCSDB), July 2014

Rating Hospital In-patient Care: TRICARE Prime Enrolled Beneficiaries TRISS Ratings of 

Hospital were consistently lower than were ratings by patients in the three comparison Health 

Systems. MHS was below the benchmark. The three comparison systems were above the 

CAHPS benchmark. Testing of statistical significance of difference was not conducted (Figure 

4.15). 

Figure 4.15 Comparison of TRISS to Three Other Health Systems for Hospital Ratings, CY13, 

Monthly

2014 MHS Review Group

Source: TRICARE Inpatient Satisfaction Survey, July 2014

Rating In-patient Hospital: TRICARE Prime Enrolled Beneficiaries 2013 TRISS Ratings of 

Recommend Hospital were consistently lower than were ratings by patients of Health System 1 

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

13-Jan 13-Mar 13-May 13-Jul 13-Sep 13-Nov

Percent Satisfied

HCAHPS Benchmark TRISS Health System 1 Health System 2 Health System 3

127
August 29, 2014 4. Quality of Care in the Military Health System

and Health System 3 and slightly lower than Health System 2 (Figure 4.16). Testing of statistical 

significance of difference was not conducted. 

Figure 4.16 Comparison of TRISS to Three Other Health Systems for Recommend Hospital, CY13, 

Monthly

2014 MHS Review Group

Source: TRICARE Inpatient Satisfaction Survey, July 2014

Summary of Findings:

1. According to the HCSDB survey, MHS beneficiaries rate their health plans higher than 

the civilian benchmark, but lower for overall health care.

2. Overall satisfaction with inpatient surgical care exceeded the civilian benchmark.

3. Overall satisfaction with obstetrics falls below the civilian benchmark, singularly 

lowering the overall inpatient scores below the civilian benchmark.

4. There is room for DoD improvement regarding the experience of care with primary care. 

PCMH-accredited sites have higher levels of satisfaction than non-accredited PCMH.

5. Data indicate that MTF enrollees are less likely than comparable civilian populations to 

see the same provider and get an appointment when they feel one is needed. 

 Recommendations Regarding Patient Satisfaction

a. MHS governance should continue to study determinants of patient satisfaction and 

develop strategies to meet or exceed civilian benchmarks in satisfaction with 

primary care and obstetrics for every MTF.

b. MHS governance should continue to guide MTFs in implementation of strategies to 

optimize PCMH operations and use of secure messaging, Nurse Advice Line (NAL), 

and other customer service tools.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

13-Jan 13-Mar 13-May 13-Jul 13-Sep 13-Nov

Percent Satisfied

HCAHPS Benchmark TRISS Health System 1 Health System 2 Health System 3

128
Military Health System Review – Final Report August 29, 2014

c. Services and DHA should continue to evaluate determinants of satisfaction with 

primary care and ensure ongoing maturation of PCMH in all MTFs.

Primary Care Manager Continuity 

The MHS implemented the Primary Care Medical Home (PCMH) model of care in order to 

improve health care quality, medical readiness, access to care, and patient satisfaction, and to 

lower per capita cost growth. PCMH is an established model for primary care, designed in part 

to improve continuity of care and to enhance the effectiveness of patient-provider 

communication. Patient centeredness refers to an ongoing, active partnership with a primary 

care physician who leads a team of professionals dedicated to providing proactive, preventive, 

and chronic care management through all stages of life. One of the core principles of the PCMH 

model is that patients have a consistent relationship with a primary care manager (PCM), which 

continues to be the driving force behind the MHS’s transformation from a system for health care 

to one supporting health. The continuous relationship between a patient and his/her provider has 

improved patient engagement and resulted in a reduction in unnecessary treatment and 

emergency room utilization. 

The direct care component relies on NCQA to evaluate whether hospitals and clinics are 

providing medical homes. NCQA evaluations focus on every aspect of PCMH delivery to 

include: access and the delivery of enhanced access; quality of care to include the use of 

evidence-based guidelines and comprehensive care; and safety to include medication 

reconciliation and continuous improvements in all aspects of care. In these evaluations the direct 

care component stands alone: 301 military clinics have achieved NCQA recognition, and the 

direct care component boasts the highest average survey scores of all health care organizations 

undergoing evaluation. By the end of 2014, all primary care practices will have sought 

recognition by NCQA with PCM continuity serving as one of its primary standards. 

The measurement of PCM continuity provides feedback to enhance the quality of care; 

consistent appointments with a PCM facilitate patient wellness and disease prevention when 

compared to discontinuous, acute, episodic care. The measure is the rate of all appointments in 

primary care (e.g., acute, routine, wellness) that are with the MTF enrollee’s assigned PCM. 

Data for this measure are continuously available from the Service to individual PCM levels 

through TRICARE Operations Center (TOC) reports drawing data from the Composite Health 

Care System (CHCS). 

Overall, PCM continuity in the direct care component increased 9.9 percent from a FY 2012 

average of 55.4 percent to a FY 2014 (July) average of 60.9 percent: baseline PCM continuity 

was 41 percent in June 2010. Increasing the level of PCM continuity was a major quality 

initiative for the three Services and NCR MD in support of PCMH implementation. Moreover, 

improved availability of a continuous relationship with a PCM was one of the top four items 

requested by patients. The initial performance target for PCM continuity was 60 percent. In 

light of the improved overall performance of the system, the target was increased to 65 percent 

for FY 2014 (see Figure 4.17). 

129
August 29, 2014 4. Quality of Care in the Military Health System

Figure 4.17 MHS Primary Care Manager (PCM) Continuity

2014 MHS Review Group

Source: TRICARE Inpatient Satisfaction Survey, July 2014

Each Service and the NCR MD increased PCM continuity during this period with Navy 

achieving the highest average in FY 2014 and the greatest rate of change from FY 2012. Though 

improving, NCR MD lags behind the Services in PCM continuity (Figure 4.18). 

Figure 4.18 Percent of Appointments where the Patient Saw their Assigned PCM – By Service

2014 MHS Review Group

Source: TRICARE Operations Center, July 2014

30%

35%

40%

45%

50%

55%

60%

65%

70%

Oct-11

Dec-11

Feb-12

Apr-12

Jun-12

Aug-12

Oct-12

Dec-12

Feb-13

Apr-13

Jun-13

Aug-13

Oct-13

Dec-13

Feb-14

Apr-14

% PCM Continuity

Percent MHS Goal

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Oct-11

Dec-11

Feb-12

Apr-12

Jun-12

Aug-12

Oct-12

Dec-12

Feb-13

Apr-13

Jun-13

Aug-13

Oct-13

Dec-13

Feb-14

Apr-14

Percent of Appointments

Month-Year

Air Force Army Navy MHS Goal NCR

130
Military Health System Review – Final Report August 29, 2014

MTF PCM continuity has increased; however, clinic and hospital continuity is higher than 

medical center continuity (Figure 4.19). Continuity tends to be lower at medical centers with 

residency programs and other direct care options such as Emergency Departments and Urgent 

Care Centers. MTF range of performance varies from 90.1 percent to 21.6 percent with a mean 

of 60.9 percent indicating the need for improvement in PCM continuity at the facility level. 

Continuity reflects the percentage of time a patient is seen by their assigned PCM when 

accessing primary care within their MTF. Every effort is made to ensure each patient’s PCM 

remains the same while the patient is enrolled to the MTF; however, PCM reassignment may be 

necessary due to the unique consequences of military service such as Permanent Change of 

Stations (PCSs), deployments, retirements, and separations. PCMH is a team-based approach to 

primary care and if PCM reassignments do occur, the MTF makes every effort to keep the 

patient with the same team that has been providing their care.

Figure 4.19 Percent of Appointments where the Patient Saw their Assigned PCM – by Facility Type

2014 MHS Review Group

Source: TRICARE Operations Center, July 2014

Overall performance on PCM continuity for CONUS and OCONUS facilities is consistent.

In summary, PCM continuity averages 61 percent; there is low variance across the direct care 

component with a median of 62 percent and an interquartile range of 56 percent to 69 percent. 

There are five positive outliers beyond two standard deviations from the mean and four negative 

outliers.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Oct-11

Dec-11

Feb-12

Apr-12

Jun-12

Aug-12

Oct-12

Dec-12

Feb-13

Apr-13

Jun-13

Aug-13

Oct-13

Dec-13

Feb-14

Apr-14

Percent of Appointments

Month-Year

Clinic Hospital Medical Center MHS Goal

131
August 29, 2014 4. Quality of Care in the Military Health System

 Recommendation Regarding Primary Care Manager Continuity

a. The PCMH Advisory Board should assess processes that affect PCM continuity at 

high-performing PCMH sites and promulgate best practices across the MHS to 

support improvement initiatives.

Site Visit Information

Seven MTF site visits were conducted to assess the onsite execution and implementation of 

DoD, Service, and NCR MD policies. Each site visit focused on key areas related to quality: 

policy and governance, leadership, quality improvement infrastructure, performance 

improvement efforts, and patient focus. Upon completion of each site visit, the team evaluated 

MTF performance against a 12-question checklist using a Likert scale that ranged from score 1 

(Not correlated) to score 5 (Exceeds). Observations from the site visits are outlined below 

(Figure 4.20).

Figure 4.20 Perceptions Among Regional Headquarters, MTF Leaders, Subject Matter Experts 

(SMEs), Staff Members and Patients During Seven MHS Site Visits, 2014

2014 MHS Review Group

Source: MHS Site Visit Survey, June - July 2014

132
Military Health System Review – Final Report August 29, 2014

Policy

Current DoD policy regarding quality of care emphasizes organizational commitment to 

performance improvement and communicates MHS goals and objectives related to quality, 

efficient, and safe patient care across the organization. However, during discussions with 

leadership, quality managers and staff it was apparent that quality practices vary among MTFs, 

with disparity in the execution and compliance of Service policies noted. Additionally, despite 

established policy, there was no consistent approach to the identification and management of 

quality issues in the purchased care component. In some MTFs the business office served as 

the point of contact for all purchased care issues, while at others the quality office served that 

role. Most MTFs forwarded all purchased care issues to the regional or overseas contractor 

representative, whereas one MTF forwarded all purchased care issues to the TRICARE 

Regional Office. The written guidance regarding the identification and management of quality 

and patient safety issues in the purchased care component was inconsistent.

Leadership

Awareness of quality initiatives and organizational performance was clearly evident at the 

executive and quality management levels, but not as evident at the staff and patient levels. MTFs 

strive to meet established national outcome benchmarks through process improvement 

initiatives. However, there is notable variability of MTF involvement in quality efforts. 

Performance initiatives were identified by leaders but staff was often unaware of its role in 

improvement efforts. 

MTFs with a multidisciplinary approach that involved provider and nursing representatives 

performed exceptionally well in their quality and data sharing efforts. At 5 of 7 sites it was 

noted that quality and performance improvement efforts were not efficiently shared across the 

MTF, which affects overall quality efforts. 

Executive leaders at all MTFs were very familiar with HEDIS®, ORYX®, Experience of Care 

and PCM continuity data, but did not have as much familiarity with NSQIP® and NPIC data. 

None of the MTFs were aware of PQI/IQI data. All MTFs were addressing National/DoD 

required benchmarks and performance measures where they were underperforming through 

measure champions, aggressive provider and clinic management staff involvement, and patient 

education/awareness initiatives. 

Quality Improvement Infrastructure

The quality improvement infrastructure was measured in relation to the following three key 

components: Resources and Staffing, Training, and Information Technology 

Resources and Staffing: Vacant positions and civilian hiring action delays were reported by 

leadership and staff at multiple MTFs. Staffing shortages include civilian and military personnel 

at all levels within the MTFs. Some MTFs reported decreasing services or shifting workload to 

purchased care. At one MTF, radiology staff shortages resulted in shifting workload to the 

network, despite the fact that MRI and ultrasound equipment were available. Staff at a number 

133
August 29, 2014 4. Quality of Care in the Military Health System

of MTFs reported working on quality initiatives and filling quality positions as an additional duty 

without training. Multiple MTF leaders and staff interviewed expressed concerns with the length 

of time it takes to hire personnel and complete in-processing requirements in a timely manner. 

Staff turnover was also identified by staff and patients as an impediment to consistent quality 

care. It was reported that MTF-wide analytic proficiency and expertise was deficient. 

Additionally, case manager personnel and services were located under different directorates. 

Case manager caseloads did not comply with current policies. For example, one MTF case 

manager had a caseload of 45 Wounded, Ill and Injured (WII) Service members, which is greater 

than the current MHS Medical Management policy of no more than 17 WII Service members per 

case manager.

Training: There is no DoD policy requirement for quality training for leaders, quality 

management personnel, or general staff. Quality staff is not required by position description to 

be certified in quality management. Most quality staff interviewed received no formal quality 

training. Recent budget constraints and changes to the DoD conference attendance policy have 

limited training opportunities. Quality staff reported not having the training or expertise to 

analyze and synthesize data to improve performance. Staff is attempting to adhere to local MTF 

policies; however there is a noticeable discrepancy in the full understanding and utilization of 

quality outcome measures and data, and staff role in implementing quality initiatives. 

Information Technology: The current information technology (IT) infrastructure and rigid 

network requirements resonate across all MTFs as problematic. MTFs expressed concerns with 

their inability to implement applications.51 Also, it was noted that there are delays for new staff 

members to gain access to the network and IT applications. Two MTFs commented on technical 

support challenges with AHLTA, Essentris, or CHCS, and concerns that this could affect patient 

care and safety (e.g., access to medical record). The process to obtain a Department of Defense 

Information Assurance Certification and Accreditation Process (DIACAP) approval for 

installation of new software is time consuming and cumbersome. One MTF experienced 

interface connectivity issues between laboratory analyzers and CHCS, lasting seven months and 

resulting in manual data entry, increasing risk for error. Three of seven MTFs were able to 

access civilian electronic health records (EHR) for enrolled beneficiaries, which enhances 

continuity and quality of care. Three MTF Emergency Departments are still using paper records.

Performance Improvement Efforts

Quality and Process Improvement (PI) initiatives are occurring throughout the MTFs. Each 

Service used specific PI methodologies (AF-AFSO21, Army-PDSA and Navy-PDCA). 

However, staff empowerment and commitment varied. 

• In several facilities it was not clear to what extent quality information and guidance was 

shared throughout the chain of command;

51 For example, peer reviews, morbidity and mortality review, radiology programs, and updating IV pump libraries.

134

 
Military Health System Review – Final Report August 29, 2014

• Sustainment plans were not always built into PI projects; 

• At a number of MTFs, leadership approved all PI projects, compared to other MTFs 

where it is more departmental-driven. 

• At most MTFs it did not appear that PI decisions involved staff at the lower level. 

Performance initiatives are top-down driven; however, frontline staff is not always aware of its

role in process improvement efforts or in the new processes resulting from the efforts. The most 

widely used improvement process was the Rapid Cycle Improvement through the use of 

successive Plan-Do-Check-Act (PDCA) type cycles. Other models used for continuous process 

improvement included Lean, Six Sigma, and Business Process Re-engineering. Management of 

quality measures varies among MTFs based on staffing, resources, data lag, and level of 

understanding and expertise. There is a lack of consistency in implementation and utilization of 

Clinical Practice Guidelines (CPG). Some of the DoD/VA CPGs are imbedded in AHLTA 

workflow forms at MTFs, but staff is not fully aware of this capability. Leadership rounds were 

being conducted at all of the MTFs visited; however, there is confusion as to the function of such 

rounds, particularly when it pertains to quality and accreditation issues. MTFs did not have 

processes or policies in place for ensuring that patients receive notification of abnormal results in 

a timely and standardized manner, unless they were critical.

Patient Focus 

Patients overall were pleased with the care they received. Patient awareness and involvement in 

processes related to quality was generally limited to their participation in surveys.52

Transparency of quality measures is limited. There was little to no patient involvement in the 

MTFs’ quality-related committees. During patient interviews, a number of patients reported not 

receiving laboratory and radiology results. None of the seven sites visited provided an 

instruction or guidance for patient notification of normal or abnormal results. All MTFs had 

policies in place that addressed critical value reporting. 

Findings from Site Visits

1. Significant variability of MTF quality efforts was noted. MTF leaders strive to meet 

established national outcome benchmarks through process improvement initiatives.

2. Despite requirements to improve transparency in current policy, efforts to improve 

transparency of quality measures at the patient level was limited and varied by MTF.

3. Current DoD policy does not require quality training for leaders, quality management

personnel, or general staff. Most quality staff interviewed reported receiving no formal 

quality training.

4. Despite established policy, there was no consistent approach to the identification and 

management of quality issues in the purchased care component. Inconsistencies were 

52 For example, ICE, APLSS, TRISS/TROSS, Service Delivery Assessment (SDA).

135

 
August 29, 2014 4. Quality of Care in the Military Health System

noted in written guidance regarding the identification and management of quality and 

patient safety issues in purchased care.

5. IT infrastructure, rigid network requirements, and technical support with clinical data 

systems were reported as problematic. Additionally, the process to obtain a Department 

of Defense Information Assurance Certification and Accreditation Process approval for 

installation of new software is time consuming and cumbersome. 

6. Leadership and staff at multiple MTFs reported vacant positions and civilian hiring action 

delays. Staff and patients identified staff turnover as an impediment to consistent quality 

care.

7. There are inconsistencies in implementation and utilization of CPGs. Some DoD/VA 

CPGs are imbedded in AHLTA workflow forms at MTFs, but staff is not fully aware of 

this capability.

8. Three of seven MTFs have access to civilian/network EHRs for enrolled beneficiaries.

9. Processes for notification of normal and abnormal laboratory and radiology results varied 

by MTF.

 Recommendations to Improve Quality from Site Visits 

a. DHA should establish clear and consistent guidelines for the CONUS TRICARE 

Regions and the OCONUS Area Offices on reporting and processing quality and 

patient safety issues identified in the purchased care component. 

b. MHS governance should work with the Services to increase the use of Clinical 

Practice Guidelines in the direct care component. 

c. MHS governance should evaluate the feasibility of DoD and TRICARE regional 

contractor collaborations/MOUs with local purchased care organizations to support 

electronic health record accessibility. 

d. MHS governance should develop processes to ensure standardized notification 

requirements for laboratory and radiology services.

Staff Town Halls

Staff town halls were conducted to assess staff knowledge and experience with quality of care 

policies and procedures. Each town hall afforded flexibility to explore issues based on the 

feedback and direction the participants desired to express, making each one unique based on 

local conditions. The theme across all sites was the varying level of successes and challenges in 

meeting quality standards. 

The staff demonstrated good understanding of quality issues within their areas and is doing 

everything within their control to take care of the beneficiary. One challenge to meeting demand 

was staffing shortages and variations in schedule management between clinics, which led to 

increased workload, and lower patient care quality and overall continuity of care. As a means to 

alleviate this, staff was willing to stay late to treat patients. They would take additional 

appointments when necessary to ensure patients were seen and not automatically sent to the 

Emergency Department. While this contributed to provider fatigue, it showed that they valued 

quality care within the MTFs.

136
Military Health System Review – Final Report August 29, 2014

The lack of continuity among providers due to workforce rotation detracts from the ability to 

deliver quality care. Frequent leadership turnover resulted in constant changes to policies and 

procedures, and affected the patient’s perception of the quality of care they were receiving. 

Workforce rotation also resulted in an ineffective transition between incoming and outgoing 

providers, causing the continuity of patient care to suffer. Staff also voiced concerns about the 

requirement to see a certain number of patients, which took precedence over delivering quality 

care to patients. There was an expectation to meet a specific patient quota, which caused staff to 

regularly work overtime to see patients and get paperwork done. Staff also commented on 

communication issues. For instance, staff indicated that there are silos that prevent good, timely 

communication across MTF departments. Staff suggested that meetings could serve as a forum 

to exchange innovative ideas and to inform all staff about key MTF initiatives, such as efforts to 

recapture. There are also concerns that patient information is not documented in such a way that 

their information could be communicated from provider-to-provider when health records are not 

readily available. Staff also expressed anxieties regarding inconsistent processes across the 

MTFs. For example, one staff member stated that the lack of standardization and hardwired 

processes has been a recurring concern that has not been adequately addressed. 

Overall, a majority of MTF staff felt that the overall culture of quality within the MHS is 

satisfactory but there is still room for improvement. TeamSTEPPS has been implemented across 

the MTFs as a means to bridge the gap in communication and eliminate any barriers to quality 

patient care. Generally, staff appeared to be passionate in their roles and strive daily to deliver 

quality care throughout the MHS.

Beneficiary Town Halls

Beneficiary town hall meetings were conducted to get a sense of the patient viewpoint in 

accessing MHS quality of care at the MTF and purchased care systems. Each meeting afforded 

participants the opportunity to voice concerns or successes, making each one unique based on the 

local conditions. 

Throughout each site visit town hall meeting, the responses ranged from ‘very satisfied’ to ‘very 

dissatisfied’, with a higher proportion of ‘dissatisfied’ responses. In some instances, 

beneficiaries spoke highly of the care they received but found it challenging to consistently 

schedule appointments with the same provider. Several patients expressed frustration that their 

PCM constantly changed, sometimes with no notification, and there were challenges seeing their 

PCM consistently within TRICARE standards. Beneficiaries were, however, very satisfied with 

RelayHealth when their doctor was an active user. 

At one location, there was a perception that the quality of care at the MTF was superior to the 

care received in the network. However, some felt the care to be lacking at the MTF and deferred 

to the network claiming shorter wait times and better provider attentiveness to patient concerns. 

Some beneficiaries found that EHRs were not accessible from direct care providers to purchase 

care providers, which negatively affected their ability to receive quality care. Several 

beneficiaries felt that thorough screenings were lacking due to the providers’ tight time deadlines 

and the strict patient quota. Patients also felt their screenings were rushed and the overall quality 

137
August 29, 2014 4. Quality of Care in the Military Health System

was deficient. Further, when participants were asked how they voiced quality concerns, several 

stated that they use customer feedback tools that were in place, such as the Interactive Customer 

Evaluation System, or communicate their concerns directly to MTF patient advocates. In many 

cases, beneficiaries indicated they were aware of and used the different mechanisms available, 

but found that once their concerns and issues were reported, they were not addressed. 

Participants also cited that patient advocates are outranked by the person whom she or he might 

receive a complaint about, and this intimidation factor often leads to an ineffective system, since 

they are reluctant to share the information up the chain. Overall, however, the participating 

beneficiaries expressed satisfaction with their care once in the system.

During interviews and town hall forums, beneficiaries expressed overall satisfaction with the 

quality of care they received, but voiced concerns that gaining access to care could be difficult. 

Town hall participants conveyed that they were unable to schedule appointments with their PCM 

or the same provider. They also stated that when they did obtain an appointment with members 

of their PCMH team, often the providers were unfamiliar with their medical history so they felt 

that they had to keep repeating their stories. Many of the concerns raised during the interviews 

and town halls were validated during the site visit walking rounds. 

Quality of Care: Overall Findings and Recommendations

Based on the analysis of available MHS quality data, there are several high-level findings and 

recommendations regarding the quality of care, as summarized here. 

1. It is clear that the MHS is dedicated to quality health care and performance improvement. 

In several areas, the MHS outperforms or is equal to national benchmarks. Other areas 

were identified for focused improvement in performance and to reduce variation in 

performance. It will be necessary to refocus the organization’s quality culture for more 

rapid and continued improvement in quality of care. The MHS Review Group 

recommends that MHS governance research and implement health care industry best 

practices of a high reliability organization to revitalize and sustain necessary cultural 

changes throughout the MHS. 

2. While comparison to national benchmarks is helpful, because of the variances inherent 

among health care systems, direct comparison between the MHS and civilian health 

systems proved challenging, with limitations in the comparative portion of the analysis. 

The MHS Review Group recommends that the MHS continue building relationships with 

civilian health systems to participate in collaboration and data sharing in order to 

facilitate more complete comparisons.

3. Under-developed MHS-level enterprise processes currently limit data standardization, 

collection, and analysis to drive system wide improvement (e.g., governance, standard 

business and clinical processes, shared services). Variation exists in the use of existing 

data to identify and prioritize objectives. The MHS Review Group recommends that the 

MHS develop and implement a performance management system that links to MHS and 

Service strategies with MHS dashboards and common systemwide performance measures 

138
Military Health System Review – Final Report August 29, 2014

to support visibility of those measures across the enterprise. The MHS should also create 

and use a MHS data analytics capability to provide analysis and actionable information to 

the Services and DHA.

4. DOD quality policy (DODI/DODM 6025.13) lacks specificity with regard to quality 

measurement and performance improvement. The MHS should update or supplement 

DoDI and DoDM 6025.13 with specific guidance on quality measurement, performance 

improvement, and requirements necessary for assessing and improving quality education 

and training.

5. While there is a significant amount of quality training occurring in the Services, there is 

no clearly prescribed quality-specific training and education by MHS policy. The DHA 

Education and Training Directorate should conduct an in-depth review and needs 

assessment of quality training to assess the efficacy of training being accomplished.

6. There are gaps in the enterprise processes to validate Service compliance with policies 

and directives disseminated from ASD(HA). The MHS Review team recommends ASD 

(HA) develop and implement a process to manage and track compliance of Services and 

DHA with applicable DoD policies and directives.

139
August 29, 2014 4. Quality of Care in the Military Health System

This page left intentionally blank.

140
Military Health System Review – Final Report August 29, 2014

5. PATIENT SAFETY IN THE MILITARY HEALTH SYSTEM

Introduction

The Military Health System (MHS) Review Group analyzed current policies, governance 

structures, education and training programs, findings from relevant internal and external reports, 

and metrics used to determine if the MHS has created a culture of safety with effective processes 

for safe and reliable care. The Agency for Healthcare Research and Quality’s (AHRQ’s) 

definition of a safety culture was used to guide this analysis: 

“The safety culture of an organization is the product of individual and group values, 

attitudes, perceptions, competencies, and patterns of behavior that determine the 

commitment to, and the style and proficiency of, an organization's health and safety 

management. Organizations with a positive safety culture are characterized by 

communications founded on mutual trust, by shared perceptions of the importance of 

safety, and by confidence in the efficacy of preventive measures.”53

Each of the Military Departments has adopted patient safety goals, as described in Appendix 5.1.

Patient Safety Governance 

In 2001, the Department of Defense (DoD) Patient Safety Program (PSP) was established 

through a congressional directive to identify and report actual and potential problems in medical 

systems and processes and to implement effective actions to improve patient safety and health 

care quality throughout the MHS. The DoD PSP is a comprehensive, centralized program with 

the goal of establishing a culture of patient safety in the MHS.

The PSP promotes a culture of safety and is designed to produce greater cross-Service sharing 

and accelerate the elimination of preventable harm. The PSP focuses on design and delivery of 

innovations and solutions to promote safe practices and advance the culture of safety, including 

education and enterprise-wide transformative approaches to drive organizational change through 

the implementation of evidence-based practices to ensure safe care for all patients. 

The Patient Safety Analysis Center (PSAC) collects, maintains, analyzes, and submits reports on 

patient safety performance metrics submitted from the MTFs. With the establishment of the 

Defense Health Agency (DHA), the PSP was integrated with Clinical Quality and Risk 

Management in the Clinical Support Division to manage, track, and analyze measures to 

establish evidence-based practices that are then disseminated for field utilization. The PSAC 

53 Available at http://www.ahrq.gov/professionals/quality-patient 

safety/patientsafetyculture/hospital/userguide/hospcult1.html. The original source is Organizing for Safety: Third 

Report of the ACSNI (Advisory Committee on the Safety of Nuclear Installations) Study Group on Human Factors. 

Health and Safety Commission (of Great Britain). Sudbury, England: HSE Books, 1993.

141

 
August 29, 2014 5. Patient Safety in the Military Health System

resides within a newly established structure, the Clinical Evaluation and Analysis Branch, which 

integrates epidemiology and surveillance for patient safety and quality analysis. Together, the 

DoD PSP and the PSAC use adverse event report-based clinical and administrative data and 

lessons learned to produce products, tools, and services designed to mitigate harm and reduce 

errors and to assist with education and training. 

The DoD PSP manages operations through the Patient Safety Improvement Collaborative 

(PSIC), which includes representatives from the Services, NCR MD, TRICARE Regional 

Offices (TROs), and the Uniformed Services University’s DoD Patient Safety and Quality 

Academic Collaborative (PSQAC). The PSQAC aims at improving clinical practice and health 

policy focused on MHS quality and safety research and education. The PSIC reports directly to 

the MHS Clinical Quality Forum in DHA. It prioritizes outcome-based patient safety targets, 

facilitates tri-Service efforts to translate evidence into practice, and coordinates standardized 

patient safety activities across the direct care component. (For Service-specific governance on 

patient safety program processes, see Appendix 5.2.) 

In 2013, MHS senior leadership accelerated the focus to reduce preventable harm and improve 

quality of services. The MHS would benefit from emphasizing the following: highly effective 

process improvement, a fully functional safety culture, engaged leadership, and the ability to 

proactively and prospectively discover and fix unsafe conditions. 

In health care, often the culture is to react after patients are harmed rather than to be proactive 

and find ways to prevent the harm. To facilitate and cultivate a more proactive organizational 

approach, the Deputy Assistant Secretary of Defense for Health Affairs chartered the Quality 

Patient Safety Risk Management Task Force (QPSRMTF) in spring 2014 with the following 

vision: 

• The MHS should strive to reduce preventable medical adverse events to zero, expect 

excellence in quality and safety across the system, and practice risk mitigation system 

wide. 

• The MHS must possess a “collective mindfulness,” that is, an ability to consistently focus 

awareness and not lose sight of factors that have the potential to cause harm, which will 

successfully transform the MHS into a high reliability organization.

Measures: Using Data to Drive Change

The PSP aggregates and analyzes event data reported to DHA and Services from MTFs, using 

various reporting systems/methods and severity ranking/harm scales to identify and report 

patient safety events. These include several iterations of Patient Safety Reporting tools, SE 

notifications, and root cause analysis (RCA).

The PSP uses data from a variety of sources to analyze and characterize patient safety 

information in order to identify systematic patterns, practices and processes that place patients at 

risk. These sources include: 

142
Military Health System Review – Final Report August 29, 2014

• The Services use a SE Notification process to report to DHA and Health Affairs. 

• RCAs are required for each SE, as defined by the DoD Manual and TJC. RCAs are indepth analyses of process and system issues, contributing factors, and identified causes 

of the reported events. 

• The PSRS, fully deployed throughout the MHS as of June 2011, allows for staff to 

directly report patient safety events. This self-reporting system also provides 

information regarding adverse drug events and patient falls, both part of the national 

Partnership for Patients effort.

• AHRQ PSIs of potential in-hospital patient safety events support initiatives aligned with 

the Partnerships for Patients (PfP).

• The Centers for Disease Control and Prevention’s (CDC’s) National Healthcare Safety 

Network (NHSN) aggregates data on reported health care-associated infections.

• The MHS administers the AHRQ Survey on Patient Safety every three years (most 

recent 2011; planned for 2015). This survey is used by organizations to survey staff on 

perceptions of leadership, staffing, teamwork, and event reporting to evaluate the culture 

of safety. 

• The Clinical Quality Forum Scientific Advisory Panel has performed a pilot Global 

Trigger Tool (GTT) Study in inpatient MTFs to evaluate this tool in relation to other 

patient safety monitoring tools currently used within the MHS. 

The recommendations for evidence-based practices derived from the data are disseminated to the 

field through PSP initiatives, education, training, and resources. 

Performance Improvement Initiatives

There are many ongoing efforts within DHA and across the Services to improve patient safety 

through performance improvement initiatives. Examples include the Partnership for Patients 

(PfP) at DHA; Patient CaringTouch System (PCTS) in the Army; Culture of Safety in the Navy; 

and reducing Surgical Site Infections in the Air Force. Details of each of these initiatives are 

found in Appendix 5.12.

Findings Related to Governance

There is variance in organizational structure for the governance of patient safety. 

 Recommendation Regarding Governance of Patient Safety

a. The Services and DHA should evaluate their organizational structure to better align 

patient safety functions within their organizations to maximize leadership visibility. 

Policy Review

DoDI 6025.13 and DoDM 6025.13

DoD Instruction (DoDI) 6025.13 (February 17, 2011) and the DoD Manual (DoDM) (October 

29, 2013)–both titled “Medical Quality Assurance (MQA) and Clinical Quality Management 

143
August 29, 2014 5. Patient Safety in the Military Health System

(CQM) in the MHS”–set requirements for patient safety programs within the MHS. Together, 

they establish policy, assign responsibilities, and provide procedures for managing the DoD PSP. 

The intent of these documents is to promote a culture of safety by eliminating patient harm 

through engaging, educating, and equipping patient care teams to institutionalize evidence-based 

safe practices. 

The TRICARE Operations Manual (TOM), Chapter 7, Section 4, requires the establishment of 

written policies to identify potential quality issues. It requires a Clinical Quality Management 

Program (CQMP) Annual Report and an analysis of the AHRQ Patient Safety Indicators (PSIs) 

to evaluate the safety of the care delivered in the network and to assess outcomes of patient 

safety programs.

54 The TRICARE Regional Office (TRO)/TRICARE Area Office (TAO) or 

Designated Provider Program Office (DPPO) provides oversight for respective contractor 

processes and compliance of the requirements in accreditation, clinical credentialing, and clinical 

quality/patient safety. 

Comparing DoDI 6025.13 for direct care providers to the requirements of the TRICARE 

contractors, it is clear that the activities required for the direct care and purchased care 

components are parallel and comparable, and meet the intent for the key functions of patient 

safety as appropriate for their role in the TRICARE program. 

Service policies are summarized below. See Appendix 5.3 for more detail.

Army Policy

The oversight of quality and patient safety has been aligned into a directorate that reports directly 

to the Deputy Commanding General for Operations, USAMEDCOM, which provides direct 

access for Army Medicine leadership to address issues in quality and patient safety. Army 

Regulation 40-68, Clinical Quality Management (CQM), establishes policies, procedures, and 

responsibilities for the administration of the Army Medical Department (AMEDD) CQM 

Program. This regulation is aligned with DoDM 6025.13 and provides the framework for 

Quality, Patient Safety and Risk Management in the AMEDD. The oversight for policy and 

standardization is delegated to the Clinical Performance Assurance Directorate (CPAD). 

Navy Policy

Navy Medicine’s patient safety policies conform to DoD policies and align with civilian 

accreditation requirements. These policies require the Navy to identify, review, and classify 

adverse events, report near misses or unsafe conditions, implement a Healthcare Resolutions 

Program, and complete proactive risk assessments. In addition, policies require every MTF to 

implement a dedicated PSP, which encourages a standardized approach to create a safer patient 

54 Such as effect on reduction of medical errors, effect on increasing patient safety, effect on health promotion and 

disease and/or injury prevention, and provider and beneficiary educational activities initiated as a result of quality 

findings.

144

 
Military Health System Review – Final Report August 29, 2014

environment, promote innovation and creativity while engaging leadership, and foster a culture 

of trust and transparency through communication, coordination and teamwork. Policies require 

the Navy to inform the patient/family of an adverse event or unanticipated outcome as soon as 

possible after the event was identified and ensure that the patient/family understand that 

discussion. To ensure compliance with these standards, both external and internal inspection 

agencies validate the MTFs’ adherence to these policies.

Air Force Policy

The Air Force Medical Service’s (AFMS’s) policy (AFI 44-119) for patient safety complies with 

DoD policy requirements, civilian accreditation standards, and aligns with current national 

patient safety standards. The policy defines patient safety program roles and responsibilities for 

executive leadership and for each health care team member rendering care. The AFMS 

complements this policy with a patient safety guidebook, which delineates process details to 

ensure uniform implementation of policy requirements. AFMS patient safety policy focuses on 

personal responsibility to identify and report near miss and actual adverse events in a timely 

fashion. Each patient safety report is analyzed to ensure that lessons are learned for performance 

improvement. Air Force policy articulates that building a culture of safety is leadership-driven 

and requires that every team member commit to the principles and practices of safe care. 

National Capital Region Medical Directorate Policy

The National Capital Region Medical Directorate (NCR MD) CQM program implements policy 

guidance, procedures, and responsibilities. Management of the NCR MD program is overseen 

by the NCR MD Quality Management Department. Revisions to the manual are managed 

collaboratively by the NCR MD Quality Management Department and the NCR MD Market 

Quality Working Group at the facility level. This management approach of the CQM program 

results in greater participation and compliance in the Quality and Patient Safety Programs by 

MTFs. 

Gaps in Policy: Findings

Although DoDM 6025.13 was published less than a year ago, staffing revisions from the original 

submission diluted the effectiveness of the Manual. The DoDM 6025.13 needs to be revised or 

supplemented with more specific guidance including input from the Service and DHA subject 

matter experts (SMEs) to improve communication, and develop a common understanding of 

definitions, taxonomies, and processes. The review identified four gaps related to policies, 

which are addressed below.

1. The self-reporting of events related to patient safety is a key concern for all health 

systems. Direct care has one central mechanism utilized to capture patient safety event 

information. Additional mechanisms are needed to ensure the capturing of all harm 

events. The reporting of events and the opportunity to learn from them in a more 

effective manner is critical. (For additional information see Patient Safety Reporting 

System, below.) 

2. The DoDM 6025.13 sentinel event (SE) definition does not currently provide sufficient 

clarity for consistent identification of sentinel events. While the definition mirrors that of 

145
August 29, 2014 5. Patient Safety in the Military Health System

The Joint Commission (TJC), there is substantial variation in interpretation at the MTF 

level. TJC has experienced similar variations in interpretation by civilian hospitals and is 

in the process of revising and expanding its definition for SE. The revised definition may 

reduce current variation across the enterprise.

3. Opportunities to partner with patients and families can help the system achieve safe, 

reliable care and exceptional experience. Engagement opportunities include formal and 

informal long-term patient/family input on specific projects and committees, as well as 

embedding the patient/family perspectives in decision making.

4. A review of DoDM 6025.13, relative to root cause analysis (RCA), provides limited 

guidance on the parameters of a quality RCA. Current RCAs vary in the analysis of 

investigations and the scope of corrective action, which makes it difficult to understand 

and learn from the event.

 Recommendations Regarding Patient Safety Policies

a. Refine DoDM 6025.13 policy to establish more than one mechanism for capturing 

harm events. 

b. Health Affairs, through the DHA Clinical Support Division, with Service 

representation, should assess the revised TJC definition of “sentinel event” and 

determine if additional guidance in the DoDM 6025.13 policy is required.

c. Health Affairs, through the DHA Clinical Support Division and Office of General 

Counsel, with Service representation, should incorporate and define appropriate 

policy for patient/family engagement to proactively include patient/family 

perspectives in MTF decision making.

d. Establish clear expectations in DoDM 6025.13 for the root cause analysis (RCA)

process. 

Review of External Reports Regarding Patient Safety 

Seventeen reports were reviewed, the most important of which is an external review performed 

by Lumetra in 2007-200855. Lumetra is an independent, nonprofit, health care consulting 

organization. The other 16 reports either had similar recommendations as or referenced the 

Lumetra Study. 

The 2008 Lumetra Study identified multiple findings, five of which remain of concern. These 

include areas lacking sufficient policies, programs, or systems within the reporting hierarchy of 

the MHS, and limitations in dissemination of potentially beneficial knowledge across the 

Services. The fifth finding, regarding leadership engagement, is addressed as a finding under 

Education and Training in this chapter. 

55 Lumetra, 2008. External Review of the DoD Medical Quality Improvement Program. Available at: 

http://tricare.mil/tma/congressionalinformation/downloads/Review%20of%20DoD%20Medical%20Quality%20Imp

rovement%20Program.pdf.

146

 
Military Health System Review – Final Report August 29, 2014

Findings Regarding Response to External Reviews

1. While alerts and advisories are disseminated from the Patient Safety Analysis Center 

(PSAC) and the Services, there is no single closed loop system to ensure documentation 

and disposition of an alert or advisory. 

2. The MHS adopted the AHRQ harm classification scale in 2010, which identifies “near 

miss” as that “which did not reach the patient.” Current policy requires 100 percent 

reporting of “near misses” in the Patient Safety Reporting System (PSRS), which is 

unattainable in any system.

3. Current processes limit the ability to exchange ideas, share lessons learned, and increase 

opportunities for systemic process improvement. There is no secure, electronic, central 

resource library to support daily operations for patient safety. There is a need for greater 

visibility of patient safety data across the organization.

4. Constraints within the resource management systems have been a barrier to authorizing 

additional federal positions. The Services maximize resources and continue to evaluate 

the appropriate mix of staff depending on resources and program needs. 

 Recommendations Regarding MHS Response to External Reports

To address the findings of external reviews, MHS governance should: 

a. Establish a system wide closed loop mechanism for documentation and disposition 

of a patient safety alert or advisory. 

b. Ensure that policy establishes attainable goals for “near miss” reporting.

c. Establish a system wide structure to fully expand internal transparency of patient 

safety information in compliance with 10 U.S.C. § 1102. 

d. DHA should conduct a business case analysis that identifies the most effective 

method for staffing the Patient Safety Program. 

Education and Training: Patient Safety Program

The PSP offers an array of education and training initiatives, programs, and products. Through 

centralized continuing education (CE) accreditation services provided by the PSP, nearly 23,000 

CE credits have been processed since 2010 for PSP training courses and on-demand learning 

events. In addition, the PSP provides the field with the latest innovations in patient safety and 

quality by offering all patient safety professionals the ability to order PSP resources for their 

facilities, receive monthly Learning Updates and eBulletins, receive PSAC publications based on 

adverse event analyses, and have virtual access to PSP resources through the Patient Safety 

Learning Center and PSP website. 

The PSP provides centralized support, products and services to build patient safety skill and 

competency, including: 1) Key PSP Initiatives (Basic Safety Manager Course; TeamSTEPPS®; 

Partnership for Patients Initiative), 2) PS Resources (Portfolio of Resources including 

publications), and 3) Recognition (Awards). 

147
August 29, 2014 5. Patient Safety in the Military Health System

(Appendix 5.5 includes an in-depth discussion of direct care and Service-specific education and 

training programs.) 

Gaps in Education and Training: Findings

1. There is no enterprise-wide integrated patient safety and quality training program to 

strengthen the development of a culture of safety and increase the ability of DoD to 

successfully engage in performance improvement efforts. 

2. Currently there is no succinct DoD patient safety resource available for executive 

leadership to effectively advance the science and practice of quality and safety within 

their organizations (recommendation from the Lumetra study). A standardized patient 

safety executive toolkit would provide medical leaders guidance for engagement and 

activation in systematic process improvement to foster a culture of patient safety. 

 Recommendations Regarding Education and Training in Patient Safety

a. Further define and standardize minimal patient safety training requirements as 

outlined in DoDM 6025.13 policy. 

b. Develop an executive leadership toolkit; this best practice guide will address integral 

areas of patient safety. 

Measures of Safety

A literature review was performed to identify PSRS used in civilian health care systems. 

PubMed was searched using the keywords: ‘Sentinel Events’; ‘Patient Safety Reporting’; 

‘Patient Safety Culture’; and ‘Root Cause Analyses.’

Existence of benchmarks for the following safety measures was assessed: 1) SEs56 stratified by 

event type, 2) patient safety reporting (distribution by degree of harm), 3) PS culture survey 

(AHRQ Hospital and Ambulatory), 4) RCAs, and 5) PSI #90 composite score. Also assessed 

was whether a national consensus or scientific evidence exists to support PSRS or other 

strategies and tools to identify and mitigate risks to patients. The TJC publishes National Patient 

Safety Goals and elements of performance, but metrics are not quantified. TJC requires that a 

RCA be performed for every SE, and outlines a “Framework for Conducting a Root Cause 

Analysis and Action Plan.” While exact adverse event reporting rates remain unknown, the 

literature generally reports that fewer than 10 percent of adverse events are reported nationally.

Myriad challenges confront PS benchmarking, with efforts relying on raising awareness to 

reduce hazards. DoD uses TeamSTEPPS®, an evidence-based teamwork collaboration and 

communication strategy developed by DoD in collaboration with AHRQ, aimed at optimizing 

performance among teams of health care professionals. Tools, such as the TapRooT®

56TJC defines an SE is an unexpected occurrence involving death or serious physical or psychological injury, or the 

risk thereof. See discussion of Measure 4 in this section.

148

 
Military Health System Review – Final Report August 29, 2014

methodology for conducting RCAs within the MHS direct care component, provide a structured 

method to analyze serious adverse events. Similar national collaboration and communication 

strategies and mechanisms are lacking. 

PSRS lack the ability to account for the influence of bias in reporting. Lack of standardized tools 

to manage PSRS information further hampers prioritization of PS efforts, nationally. Assessing 

the impact of PS initiatives and strategies requires assessment of generally accepted, rigorous, 

standardized, and practical measures of adverse events and near misses. Current systems lack 

quantitative methods to assess whether PS improves as the result of a targeted initiative. 

Additionally, scarce resources exist to evaluate what works and, if so, at what cost. The role of 

leadership in promoting the culture of patient safety in health care is extremely valuable; 

however, quantifying that value in improvements in PS is difficult.

Additionally, the MHS Review Group reviewed and analyzed data for the direct care component 

with the three comparative health systems. The three measures compared were: PSI #90, 

NHSN, and the AHRQ Survey on Patient Safety Culture.

Measures within Direct Care settings

Patient Safety Culture Survey 

The AHRQ Survey on Patient Safety Culture is a validated measurement tool offered by the 

MHS direct care component on three occasions over the past 10 years: 2005, 2008, and 2011

(See Appendix 5.6). This voluntary survey is administered at the MTF levels and is designed to 

help hospitals assess the culture of safety at the local level by collecting staff opinions and 

perceptions of leadership, communication, reporting and staffing/teamwork. Due to the local 

nature of culture, information is displayed in aggregate. 

AHRQ has established the Hospital Survey on Patient Safety Culture Comparative Database as a 

central repository for survey data from hospitals that have administered the AHRQ Patient Safety 

Culture Survey Instrument, allowing comparison with other hospitals. 

The Hospital Survey on Patient Safety Culture (HSOPS) was administered in 2005 and 2008 

across MHS direct care facilities. The Medical Office Survey on Patient Safety was conducted 

in Air Force ambulatory (only) facilities in 2011; thus, Air Force ambulatory sites do not have 

three comparative data points. In 2011, all other inpatient and outpatient facilities used the 

HSOPS survey. This survey assesses 12 dimensions of the culture of safety, presented in Table 

5.1. The dimensions emphasized in bold are the areas of special consideration for this review to 

gauge the adoption of a culture of safety. Table 5.2 shows direct care data for the HSOPS survey 

conducted in 2005, 2008, and 2011.

In order to compare the direct care component and Health System 3 results from the Hospital 

Survey on Patient Safety Culture, items were recoded according to the AHRQ methodology. 

These recoded items were then grouped into 12 dimensions and matched to the AHRQ survey 

used by both Systems. 

149
August 29, 2014 5. Patient Safety in the Military Health System

Table 5.1 HSOPS Dimensions

Dimensions

D1: Management Support for Patient Safety D2: Supervisor/Manager Expectations and 

Actions Promoting Patient Safety

D3: Organizational Learning – Continuous 

Improvement D4: Non-punitive Response to Error/Mistakes 

D5: Feedback and Communication about Error D6: Frequency of Events Reported 

D7: Communication Openness D8: Teamwork within Units 

D9: Teamwork across Units D10: Handoffs and Transitions 

D11: Staffing D12: Overall Perception of Patient Safety 

Dimensions in bold are the specific areas of focus of this report in order to gauge the adoption of a culture of safety.

2014 MHS Review Group

Source: Final MHS Overall Culture Survey Final Report, January 2013

Table 5.2 Direct Care Component HSOPS Results: Average Percent Positive Responses across

Dimensions

DoD 

Year

Response 

Rate D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12

2005 54% 71% 72% 68% 44% 64% 60% 61% 75% 59% 47% 45% 66%

2008 58% 72% 73% 69% 44% 63% 62% 61% 75% 59% 49% 46% 66%

2011 43% 72% 73% 67% 42% 62% 64% 61% 75% 59% 49% 48% 66%

2011 

AHRQ 52% 72% 75% 72% 44% 64% 63% 62% 80% 58% 45% 57% 66%

Dimensions in dark gray columns are the specific areas of focus of this report in order to gauge the adoption of a 

culture of safety

2014 MHS Review Group

Source: Final MHS Overall Culture Survey Final Report, January 2013

The direct care component as a whole showed limited improvement between 2008 and 2011. 

Two dimensions showed improvement between 2008 and 2011; D6 “Frequency of Events 

Reported” and D11 “Staffing.” No dimensions met AHRQ’s “practical significance” definition 

of a +/- 5 percent change (See Appendix Table 5.6-1). Although the perception of respondents is 

that events are reported frequently, the number of respondents who actually reported an event is 

just more than 25 percent (one of the six questions behind the D6 aggregate). This lags behind 

the AHRQ reference population, where 46 percent of respondents had reported an event. Table 

5.3 contains direct care percent positive responses across the five areas of special consideration 

for 2008 and 2011 survey years, as well as the 2011 AHRQ Reference response proportions 

(using 2011 data). All five domains were lower than the AHRQ comparison positive response 

rate; of note, Organizational Learning, Teamwork within Units, and Staffing were below the 

AHRQ practical significance change of 5 percent. 

150
Military Health System Review – Final Report August 29, 2014

Table 5.3 Average Percent Positive Responses Across Dimensions

DoD Year Response

Rate

Supervisor/

Manager 

Expectations and 

Actions 

Promoting Patient 

Safety

Organizational 

Learning – 

Continuous 

Improvement

Non-punitive 

Response to 

Error/

Mistakes

Teamwork 

in Units Staffing

2005 54% 72% 68% 44% 75% 45%

2008 58% 73% 69% 44% 75% 46%

2011 43% 73% 67% 42% 75% 48%

Decrease/ 

Flat/

Increase

↓ → ↓ ↓ → ↑

AHRQ 2011 52% 75% 72% 44% 80% 57%

Compare to 

AHRQ ↓ ↓ ↓ ↓ ↓ ↓

2014 MHS Review Group

Source: Final MHS Overall Culture Survey Final Report, January 2013

Based on the comparison of 2008 and 2011 survey results, only one of the five focused 

dimensions showed improvement: D11 Staffing, which contains questions regarding crisis 

mode, use of temporary workers, hours, and workload. The perception of staffing lags 

significantly behind civilian health care systems. Response rate is also an indicator of the 

importance placed on the culture of safety. The response rate dropped by 15 percent in 2011 

compared to 2008. All other dimensions remained flat from 2008 to 2011. 

Facilities should be confident using the survey information as a data source for gauging patient 

safety culture. Because the survey unit of analysis is the organization and not the individual, 

survey results remain relevant over time. Use of the survey data allows facilities to view trends 

in order to determine targeted initiatives. Given the use of the survey across the organization, the 

data provide insight into the importance and adoption of a culture of safety within the direct care 

component as a whole and a comparison to civilian hospital counterparts. 

External Health System Comparison Results

Differences in percent positive values were tested for significance using a t-test (assuming nonordinal data), and Health System 3 scores were significantly higher on the following dimensions: 

Supervisor Expectations and Actions, Organizational Learning/Continuous Improvement, 

Feedback and Communication about Error, Teamwork within Units, Teamwork Across Units, 

Handoffs and Transition, Staffing, and Overall Perceptions of Patient Safety. There were four 

domains where direct care results are similar to Health System 3 and the AHRQ overall. 

Frequency of Events reported is an area that direct care had a higher percent positive response 

151 
August 29, 2014 5. Patient Safety in the Military Health System

than both Health System 3 and the AHRQ overall. Non-punitive Response to Error/Mistakes 

appears to be a domain with which all systems struggle. The AHRQ 2011 overall percent 

positive result was 44 percent, direct care was 42 percent, and Health System 3 was slightly 

higher at 45.3 percent; again, not significantly higher (see Table 5.4 and Appendix Table 5.6-2).

Table 5.4 HSOPS Percent Positive Results for Comparing Direct Care 2011 Results to Health 

System 3 Survey

Survey domain

DoD culture results: 

"Same" 

"Performs better" 

"Needs 

improvement"

2011 DoD Patient 

Safety Culture 

Percent Positive 

results

2012 System 3 

Hospital Survey on 

Safety Culture

MHS Review Team Focus areas from the

Hospital Survey on Patient Safety Culture

D2: Supervisor/Manager 

Expectations and Actions 

Promoting Patient Safety

Needs improvement* 73% 77.8%

D3: Organizational Learning – 

Continuous Improvement 

Needs improvement** 67% 78.8%

D4: Non-punitive Response to 

Error/Mistakes

Same 42% 45.3%

D8: Teamwork within Units Needs improvement** 75% 86.8%

D11: Staffing Needs improvement** 48% 59.5%

Other Domains of the Hospital Survey on Patient Safety Culture

D1:Management Support for 

Patient Safety 

Same 72% 76.7%

D5: Feedback and 

Communication about Error 

Needs improvement* 62% 68.2%

D6: Frequency of Events 

Reported 

Same 64% 62.3%

D7: Communication 

Openness 

Same 61% 63.0%

D9: Teamwork across Units Needs improvement** 59% 69.0%

D10: Handoffs and 

Transitions 

Needs improvement** 49% 56.4%

D12: Overall Perception of 

Patient Safety 

Needs improvement** 66% 74.5%

*Statistically significant, p<0.05

**Statistically significant, p<0.01

2014 MHS Review Group

Source: Final MHS Overall Culture Survey Final Report, January 2013

152 
Military Health System Review – Final Report August 29, 2014

External Health System Comparison: Limitations to Interpretation

These results should be interpreted with caution, as direct comparisons of survey results are 

inherently problematic. In both the direct care component and Health System 3 data, it is unclear 

what population was sampled in the hospital. Additionally, it is unclear which type of sampling 

was used (e.g., random sample, census, stratified random sample). Finally, response rates are 

unknown for Health System 3; although they are given for direct care, it is unclear if there were 

any non-response weights applied to the data, which may significantly affect the scores. In 

summary, further review of the culture survey data would be required to make any definitive 

comparisons between direct care and System 3.

Findings Regarding a Culture of Safety

1. Direct care results indicate a lower percentage of positive responses in the adoption of a

culture of safety compared to AHRQ average national score with limited improvements

observed over time and less favorable position when compared to the civilian averages (7

of 12 dimensions with lower scores; but only 3 dimensions meet AHRQ criteria for

practical significance). A declining survey response rate over 3 iterations may indicate a

lower level of engagement and emphasis in patient safety overall. Wide variation is

found in scores across MTFs. Hospitals across the direct care component do not appear

to be as similar as expected for an integrated delivery system (data not presented). In the

external health system comparison, there are eight domains with results lower and four

domains with results similar to Health System 3.

2. Staffing consistently ranked as one of the lowest scoring across three surveys.

Qualitative comments indicate concerns about clinical experience, clinical oversight,

guidance, and access to resources required to perform duties.

 Recommendations to Improve a Culture of Patient Safety

a. MHS senior leadership must determine safety culture expectations and set targets

based on opportunities.

PSI #90 Composite for the Military Health from CY 2010-2013 

The PSIs are a set of measures developed by AHRQ that enable health care organizations to 

screen for adverse events that may have occurred during the process of health care delivery.

57 

Since it is believed that these events are preventable at the system and provider levels, 

improvement can be assessed through ongoing monitoring. Patient Safety for Selected 

Procedures Composite – (PSI #90), the focus of this analysis, is a consensus-based aggregation 

of select PSIs for eight frequently observed patient safety problems in the inpatient setting (see 

Appendix 5.7). These indicators include pressure ulcer (PSI #03), iatrogenic pneumothorax (PSI 

#06), infection due to medical care (PSI #07), postoperative hip fracture (PSI #08), postoperative 

pulmonary embolism or deep vein thrombosis (PSI #12), postoperative sepsis (PSI #13), 

57 See http://www.qualityindicators.ahrq.gov/modules/psi_overview.aspx.

153
August 29, 2014 5. Patient Safety in the Military Health System

postoperative wound dehiscence (PSI #14), and accidental puncture or laceration (PSI #15). The 

eight measures selected were endorsed by the National Quality Forum (NQF) in 2009 and are 

weighted to reflect NQF criteria for endorsement.

58 Of note, PSI #90 was not publicly reported 

on Hospital Compare59 during the 2010 to 2013 period, and DoD did not aggregate and use the 

PSI #90 composite for provider or enterprise-level quality improvement. The Centers for 

Medicare & Medicaid Services (CMS) intend to publish PSI #90 composite to Hospital Compare 

in 2014.60

For comparisons, measures of central tendency (mean/median) and dispersion of the PSI #90 

composite were estimated at 95 percent confidence intervals for both direct care data and each 

health system. Variance of the mean PSI #90 Score across systems was compared with followup testing for significant differences. 

This comparison was further informed by assessing performance of the direct care component 

and three external health systems relative to the Healthcare Cost Utilization Project (HCUP) 

State Inpatient Database reference population for each year, assuming a similar case mix for a 

given year. 

Relative Performance of Direct Care

Although the trend in the PSI #90 is informative, comparisons against reference populations or 

the national external benchmark provide an assessment of relative performance. For PSI #90, 

relative performance of the direct care component was assessed by comparing its data to the 

AHRQ reference population61 and the three CMS national achievement thresholds62 with three 

possible outcomes against the two benchmarks: direct care “outperformed,” performed the 

“same as,” or “underperformed” the benchmark AHRQ reference population or CMS national 

achievement threshold. 

58 See 

http://www.qualityindicators.ahrq.gov/Downloads/Modules/PSI/V43/Composite_User_Technical_Specification_PSI

_4.3.pdf.

59 Hospital Compare is a CMS website used to find hospitals and compare quality of care. Available at: 

www.medicare.gov/hospitalcompare.

60 See 

https://qualitynet.org/dcs/ContentServer?c=Page&pagename=QnetPublic%2FPage%2FQnetTier3&cid=1228695321

101.

61 Reference population is created from the AHRQ-sponsored Healthcare Utilization Project State Inpatient 

Database, which is home to the most extensive inpatient discharge abstracts from participating States.

62 National Achievement Thresholds for Performance for PSI #90 Composite .68(2010, 2011), .61(2012) and 

.62(2013)

154
Military Health System Review – Final Report August 29, 2014

DHA and Service-Level Trend Analysis 

The PSI #90 composite was reviewed to assess for trends in the direct care component. At the 

DHA and Service levels, statistically significant decreases in the PSI #90 composite were 

observed from CY 2010 to CY 2013 using Ordinary Least Squares (OLS) Regression (p<.001). 

Decreasing composite scores equate to positive improvement. For direct care, the PSI #90 

decreased by an estimated 2.8 percent per quarter, while the PSI #90 for the Army, Navy, and 

Air Force decreased by 1.4 percent, 3.4 percent, and 0.1 percent, respectively. 

Military Treatment Facility Analysis 

As shown in Figure 5.1, performance reflective of the direct care component overall, the 

observed decrease in PSI #90 corresponded to an annual increase in the percentage of MTFs that 

either performed the same as or outperformed the AHRQ reference population from 2010 to 

2013. On an annual basis, an average of 87 percent of MTFs performed the same as or 

outperformed the AHRQ reference population (See Appendix Table 5.7-1). At the Service level 

similar trends were observed with no statistically significance differences observed among the 

Services in the average number of MTFs that performed the same or outperformed the AHRQ 

reference population. 

Figure 5.1 MTF Performance versus Reference Population, CY10 – CY13

2014 MHS Review Group

Source: Military Health System Population Health Portal (MHSPHP), July 2014

In Figure 5.2, when compared to CMS national achievement threshold in the same period, 72 

percent of MTFs performed the same as this CMS benchmark for the CYs 2010 to 2013. The 

PSI #90 rate increased from 64 percent in 2010 to 75 percent from 2011 to 2012 and dropped to 

73 percent in 2013. A similar consistent overall increase was noted for all Services. A 

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2010 2011 2012 2013

% of MTFs

Calendar Year

% Outperforming

% Same As

% Underperforming

155
August 29, 2014 5. Patient Safety in the Military Health System

significant difference between the Services was observed for the Navy compared to the Air Force 

related to a higher annual percentage of Navy MTFs performing the same as the national 

achievement threshold. No difference was observed in pairwise comparisons between the Army 

and the Air Force and the Army and the Navy (p<.05) (One way Analysis of Variance; p=.031). 

Figure 5.2 MTF Performance versus National Benchmark Rate, CY10 – CY13

2014 MHS Review Group

Source: Military Health System Population Health Portal (MHSPHP), July 2014

Medical Center (MEDCEN) Analysis 

From 2010 to 2013, 13 MEDCENs were evaluated for performance using PSI #90. 

Approximately two-thirds of MEDCENs performed the same as or outperformed the AHRQ 

reference population; one-third of MEDCENs performed the same as the national benchmark 

rate. There was an increase in the proportion of MEDCENs performing the same as the average 

national benchmark rate from 2010 to 2013. Of note, four MEDCENs (San Antonio Military 

Medical Center [SAMMC] – Ft. Sam Houston; William Beaumont Army Medical Center 

[WBAMC] – Ft. Bliss; 60th Medical Group [MED GRP] – Travis; Naval Medical Center 

[NMC] Portsmouth) outperformed the reference population at least once during the four-year 

observation, with nine performing the same as the reference population and two MEDCENs 

(88th MED GRP – Wright Patterson; Madigan Army Medical Center – Ft. Lewis) 

underperforming the reference population across the observation period. Even the two relatively 

underperforming MEDCENs demonstrated an improvement from 2010 to 2013. While there 

was variation in the performance of MEDCENs as compared to two different benchmarks, there 

was an overall trend of improvement. 

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2010 2011 2012 2013

% of MTFs

Calendar Year 

% Same As

% Underperfoming

156
Military Health System Review – Final Report August 29, 2014

Hospital-Level Analysis 

From 2010 to 2013 all direct care hospitals (44) across all Services performed the same as the 

reference population, with 86 percent performing the same as the national achievement 

threshold. No statistically significant differences were observed among the Services.

OCONUS MTF Analysis 

From 2010 to 2013, 100 percent of outside the continental United States (OCONUS) MTFs 

performed the same as the AHRQ reference population while 93 percent performed the same as 

the national benchmark rate. No statistically significant differences were observed among the 

Services.

External Health System Comparison Findings

PSI #90 composite was compared across all three health systems on a calendar year-to-calendar 

year basis where possible. Each health system provided point estimates for the PSI #90 

composite for a varying number of hospitals within their respective systems and for different 

time periods, which in some instances permitted the same time period to be compared. 

The PSI #90 composite for the direct care component and its associated measures of dispersion 

overlapped all three health systems for all periods observed (see Figures 5.3 and 5.4). Analysis 

of variance among all four systems demonstrated no differences between the direct care 

component and other health systems (one-way analysis of variance [ANOVA]; p<.05; p=0.000; 

all confidence intervals for post hoc pairwise comparisons included 0.) Performance relative to 

the reference population, assuming a similar case mix, was also no different across systems. The 

direct care component and one of the other systems had at least one outlier. 

External Health Systems Data: Limitations

Direct care facilities: PSI #90 data using inpatient direct care data (Standard Inpatient Data 

Record) from the DoD Data Repository. Data provided included PSI #90 composite scores using 

the NQF-endorsed, 8-indicator composite using present on admission (POA) weighted estimates. 

• System 1: Provided calendar year (CY) 2012 PSI #90 calculated scores for 14 facilities.

Information on weighting using POA was not provided.

• System 2: Provided CY 2013 PSI #90 calculated scores for three facilities. Information

on weighting using POA was not provided.

• System 3: Provided CY 2011, CY 2012 and CY 2013 PSI #90 calculated scores for 23

facilities.

• However, potential quality issues with the CY 2012 and CY 2013 data precluded use for

comparisons. Information on weighting using POA was not provided.

157
August 29, 2014 5. Patient Safety in the Military Health System

Figure 5.3 Boxplot of PSI #90 Composite: Direct Care Relative to Systems 1, 2, 3

2014 MHS Review Group

Source: Military Health System Population Health Portal (MHSPHP) and External Health Systems, June - July 2014

Figure 5.4 Interval Plot of PSI #90 Composite by System and Time Period 

2014 MHS Review Group

Source: Military Health System Population Health Portal (MHSPHP) and External Health Systems, June - July 2014

2.00

1.75

1.50

1.25

1.00

0.75

0.50

External Health System and Tim e Period

P

SI #

9

0

C

o

m

p

osite 

 

1.2

1.1

1.0

0.9

0.8

0.7

0.6

System and Time Period

P

SI #

9

0

C

o

m

p

osite

The pooled standard deviation was used to calculate the intervals.

 

 

158
Military Health System Review – Final Report August 29, 2014

External Health System Analysis Limitations

A difference in the number of facilities for which information was provided limits the precision 

of the calculated PSI #90 confidence interval for one of the health systems. The time periods 

provided by the external health systems varied, however comparison was enhanced by matching 

the direct care results to each of the time periods provided by the external health systems. Upper 

and lower confidence limits for the PSI #90 estimates were not available at the facility or system 

level. Although ANOVA is considered to be reasonably robust against assumptions of nonnormality, one health system’s data (Health System 3) were not normally distributed due to the 

small sample size provided. This limits the conclusions that can be drawn from this system.

Findings Regarding Use of PSI #90 in the MHS

1. Overall, the majority of MTFs perform the same as both the AHRQ reference population

and the CMS national achievement threshold, with hospitals performing more favorably

than MEDCENs and rare differences among Services observed. Significant differences

were noted in relative performance of the MTFs when comparing direct care data to the

AHRQ reference population and the CMS national achievement threshold. Although

some of the direct care population is likely to be similar to the Medicare fee-for-service

population, it is unclear how comparable DoD beneficiaries are to this population as it

relates to the national achievement threshold rate. The AHRQ reference population is

from the Healthcare Utilization Project State Inpatient Database (SID), which includes a

wider range of ages for patients as opposed to only Medicare eligible fee-for-service

patients.

2. At the system level, when matched to compare the same time periods, no statistically

significant differences were observed between the mean PSI #90 point estimates of the

direct care component (2011, 2012, and 2013) and all three external health systems.

3. Relative to the reference population, the direct care component performed the same as the

reference population, which was also observed for two of the three health systems. Only

one health system (Health System 1) outperformed the reference population (assuming a

similar case mix) across their facilities.

4. Although the DoD is familiar with PSIs, the aggregated PSI #90 composite has not been

used by the Services.

 Recommendation Regarding Use of PSI #90 in the MHS

Consider PSI #90 composite utilization as a component of a comprehensive safety 

measure set within the MHS and develop an education plan to support its 

implementation. 

Healthcare-Associated Infections, CY 2010 to 2013

The National Health Safety Network (NHSN) is a surveillance system operated by CDC that 

provides health care facilities with information and tools to manage and improve quality with 

159
August 29, 2014 5. Patient Safety in the Military Health System

respect to healthcare-associated infections (HAI).63 All inpatient MTFs participate in 

Partnership for Patients (PfP), a nationwide approach to improving the safety and quality of care, 

which includes HAIs as a measure of performance.

HAI occurring in medical/surgical intensive care units (ICU) have well accepted external 

benchmarks for comparison. MTFs with Med/Surg ICUs currently track the measure by 

participating in NHSN. The review and analysis compared direct care performance across three 

measures by each of the designated ICU types (CY 2010 to 2013): Central Line-Associated 

Bloodstream Infection (CLABSI), Catheter-Associated Urinary Tract Infections (CAUTI), and 

Ventilator Associated Pneumonia (VAP).

Two categories of Med/Surg ICUs were reviewed for this analysis using CDC criteria for ICU 

classification: Major Teaching, and Other, <15 ICU beds. The major teaching hospital group 

includes (7) = Madigan AMC, Brooke (BAMC), Tripler AMC, Travis AFB Hospital, Walter 

Reed, NMC Portsmouth, and NMC San Diego. There were 17 in the second group (Other, <15 

ICU beds facilities). Some MTFs were excluded due to insufficient data.

Two external measures generated by the NHSN program were used to assess relative 

performance. The first measure is based on the CDC practice of using the 90th percentile to 

determine whether a hospital is a HIGH outlier (higher infection rate). CDC further interprets 

performance at this benchmark to mean that 90 percent of the hospitals had lower rates and 10 

percent of the hospitals had higher rates (at the 90th percentile). The second measure to evaluate 

hospitals is a pooled mean of all respective ICU types to compare relative performance. The 

analysis attempted to answer three questions: 

• How well are participating MTF ICUs performing compared to the civilian sector?

• Are any MTFs underperforming (HIGH outliers > 90th percentile)?

• Are any MTFs outperforming (below 25th percentile)?

Analysis and Observation by ICU and Infection Types 

Catheter-Associated Urinary Tract Infections (CAUTI):

• Data collection reporting to NHSN became a requirement in 2012.

• Reflects the largest volume (in direct care component) of eligible device days of reported

HAIs.

• Direct care Med/Surg ICUs demonstrate the following percentiles of performance relative

to similar category ICUs nationwide (see Table 5.5):

o Major Teaching Hospitals

 1 (14 percent) ICU (81st MED GRP – Keesler) outperformed the 25th

percentile with 6 (86 percent) performing between the 25th and 75th

percentiles. No High Outliers identified.

63 http://www.cdc.gov/nhsn/

160
Military Health System Review – Final Report August 29, 2014

o Other Hospitals with less than 15 ICU Beds

 8 (44 percent) ICUs (633rd MED GRP – Langley-Eustis; 673rd MED

GRP – Elmendorf; 96th MED GRP – Eglin; 99th MED GRP –

O’Callaghan; Evans Army Community Hospital [ACH] – Ft. Carson;

Naval Hospital [NH] Camp Pendleton; NH Jacksonville; NH Okinawa)

outperformed the 25th percentile with 8 (44 percent) performing between

the 25th and 75th percentiles. Two (11 percent) High Outliers

(underperforming) identified (Dwight David Eisenhower Army Medical

Center [DDEAMC] – Ft. Gordon; WBAMC – Ft. Bliss).

Table 5.5 Direct Care CAUTI by ICU Type, for Total Period, CY10 – CY13

MED SURG ICU

<25th percentile 

(Out performance)

25th and 75th

Percentile

High Outliers >90th

percentile 

(May Need Improvement)

Major Teaching 1 (14%) 6 (86%) 0 

Other Hospitals, <15 

ICU beds 8 (44%) 8 (44%) 2 (11%)

2014 MHS Review Group

Source: DoD – CDC’s National Healthcare Safety Network (NHSN), FY12 Q1 – FY14 Q2, June 2014

Central Line-Associated Bloodstream Infection (CLABSI):

At the direct care level, CLABSI reflects the next largest category of eligible infection 

surveillance volume (measured in device days) (see Table 5.6). 

• Med/Surg ICUs have at least 24 MTFs actively participating in data reporting visible to

DHA (7 major teaching hospitals and 16 other hospitals).

• Major Teaching Hospitals

o 3 (43 percent) ICUs (81st MED GRP – Keesler; NMC San Diego; Tripler AMC)

outperformed the 25th percentile with 3 (43 percent) performing between the 25th

and 75th percentiles and 1 (14 percent) identified as a High Outlier

(underperforming) (60th MED GRP – Travis).

• Other Hospitals with less than 15 ICU Beds

o 3 (19 percent) ICUs (673rd MED GRP – Elmendorf; Carl R. Darnall AMC

[CRDAMC] – Ft. Hood; Ft. Belvoir Community Hospital [FBCH]) outperformed

the 25th percentile with 10 (62 percent) performing between the 25th and 75th

percentiles and 3 (19 percent) High Outliers (underperforming) identified (88th

MED GRP – Wright Patterson; Blanchfield ACH – Ft. Campbell; NH

Jacksonville)

161
August 29, 2014 5. Patient Safety in the Military Health System

Table 5.6 Direct Care CLABSI by ICU Type, for Total Period CY10 – CY13

MED SURG ICU <25th percentile

(Outperformance)

Between 25th and 

75th Percentile

High Outliers >90th 

percentile 

(May Need Improvement)

Major Teaching 3 (43%) 3 (43%) 1 (14%)

Other Hospitals, 

<15 ICU beds

3 (19%) 10 (63%) 3 (19%)

2014 MHS Review Group

Source: DoD – CDC’s National Healthcare Safety Network (NHSN), FY12 Q1 – FY14 Q2, June 2014

Ventilator Associated Pneumonia (VAP):

At the direct care level, VAP reflects the smallest category of eligible infection surveillance 

volume (measured in device days) (see Table 5.7). 

• VAP is no longer being tracked as VAP but rather as Ventilator Associated Events

(VAE). Direct care MTFs will follow the standard set by the CDC for VAE upon its

release.

• Major Teaching Hospitals

o No ICUs outperformed the 25th percentile with 6 (86 percent) performing

between the 25th and 75th percentiles and 1 (14 percent) High Outlier

(underperforming) identified (NMS Portsmouth).

• Other Hospitals with less than 15 ICU Beds

o 5 (36 percent) ICUs outperformed (633rdd MED GRP – Langley-Eustis; 673rd

MED GRP – Elmendorf; 99th MED GRP – O’Callaghan; Blanchfield ACH – Ft.

Campbell; Evans ACH – Ft. Carson) the 25th percentile with 6 (43 percent)

performing between the 25th and 75th percentiles. Three (21 percent) High

Outliers (underperforming) identified (88th MED GRP – Wright Patterson;

FBCH; DDEAMC – Ft. Gordon).

Table 5.7 Direct Care VAP by ICU Type, for Total Period CY10 – CY13

MED SURG ICU <25th percentile 

(Outperformance)

Between 25th and 

75th Percentile

High Outliers >90th 

percentile

(May Need Improvement)

Major Teaching 0 6 (86%) 1 (14%)

Other Hospitals, <15 

ICU beds 5 (36%) 6 (43%) 3 (21%)

2014 MHS Review Group

Source: DoD – CDC’s National Healthcare Safety Network (NHSN), FY12 Q1 – FY14 Q2, June 2014

External Comparison: Health Care-Associated Infections

The MHS Review Group was able to compare these same measures with all three external health 

care systems, although there were limitations (see Table 5.8).

162 
Military Health System Review – Final Report August 29, 2014

Limitations of Comparison System

Health System 1 summary of performance was based on 12-month rolling data and calculated as 

an evenly weighted pooled mean. CAUTI and CLABSI rates are associated with ICUs. Health 

System 1 VAP rate may not be associated with ICUs. Health System 2 supplied data on 

infections for up to four years. Of the inpatient unit data provided, only two appear to 

correspond to ICUs. Data show the majority of infections identified (and device days) are 

largely outside of ICU designated units. Health System 3 VAP data included quarterly figures 

and rates, with no data at the facility or unit level. It is unknown whether the VAP data 

represents ICUs, non-ICUs, or both. 

In summary, despite data comparison limitations, the external system data suggest the 

following:

• The direct care component should consider tracking infection rates at the unit level

beyond ICUs.

• ICU CLABSI rates present an opportunity for improvement.

• ICU CAUTI rates may be comparable if ICU case-mix matches those of the external

systems. (See Table 5.8.)

Table 5.8 DoD Direct Care and Civilian Health Care Systems HAI Rates

DoD HS1** HS2 HS3

CAUTI 3.28 ICU 1.49 ICU 2.44 3.82 ICU

0.69 non-ICU

CLABSI 2.07 ICU 0.58 ICU 1.25 0.59 ICU

VAP 4.57 ICU 0.90 0.86 1.68

Green font indicates that the System outperformed DoD 

Red font indicates that the Health System underperformed DoD 

HS2 - infection data for CY12Q1-CY13Q4, July 2014 

HS3 - infection data for ICU infections CY10Q1-CY14Q1, July 2014 

*Direct comparisons by ICU type could not be made consistently due to the provision of a range of ICU types by external health

systems 

**System 1 rates reflect 12-month rolling data. 

2014 MHS Review Group

Source: DoD - CDC’s National Healthcare Safety Network (NHSN), FY12Q1-FY14Q2, June 2014

Findings Regarding Use of the NHSN Metrics 

1. For CAUTI:

o Major Teaching Facilities: The majority of ICUs fell between the 25th and 75th

percentiles with one high performer but no underperformers.

o ICUs with less <15 beds: The majority were either met or outperformed with two

underperformers.

163
August 29, 2014 5. Patient Safety in the Military Health System

2. For CLABSI:

o Major Teaching Facilities: Most ICUs fell within the normal percentile range with

one underperformer.

o ICUs with less <15 beds: The majority of ICUs fell between the normal percentile

range with three each underperformers and outperformers.

3. For VAP/VAE:

o Major Teaching Facilities: Most ICUs fell within the normal percentile range with

one underperformer.

o ICUs with less <15 beds: The majority fell within the normal percentile range

with five outperformers and three underperformers.

4. There is no comprehensive plan to standardize requirements for monitoring devicerelated infections.

See Appendix 5.8 for graphical representation of NHSN findings. 

 Recommendations Regarding Use of NHSN Metrics

a. The Infection Prevention and Control Panel should review variance in performance

in accordance with the PfP Implementation Guides for CLABSI and VAP/VAE.

b. The Infection and Prevention Control Panel should develop a comprehensive plan to

standardize requirements for monitoring device-related infections.

Sentinel Event (SE) Reporting 

According to TJC, a sentinel event (SE) is an unexpected occurrence involving death or serious 

physical or psychological injury, or the risk thereof. Serious injury specifically includes loss of 

limb or function. The phrase, “or the risk thereof” includes any process variation for which a 

recurrence would carry a significant chance of a serious adverse outcome.

64 If SEs meet the 

qualifying criteria, they must be reported within 24 hours of discovery by the Services using the 

SE Notification process. Designated DHA staff is notified through the SE Notification process. 

TJC collects voluntary SE report information and provides summaries of SEs reviewed in 

periodically published reports. SE reporting represents one of the least comparable areas of 

patient safety because SE reporting is mandated within all MTFs and is primarily voluntary in 

civilian systems. Because the reporting is voluntary, the data are not considered epidemiologic 

data sets and no conclusions should be drawn about the actual frequency of events or trends over 

time.

64 See Appendix 5.9. See The Joint Commission. (Mar 2013). Comprehensive Accreditation Manual for Hospitals: 

Sentinel Event (SE) (Update 1). Oakbrook Terrace, IL: The Joint Commission.

164
Military Health System Review – Final Report August 29, 2014

As seen in Patient Safety Culture Survey results, the small improvements in reporting events (62 

percent average positive score for 2005 to 2011 in D6 Frequency of Events Reported) may be 

curtailed by an underlying fear of retribution for reporting as supported by the consistently low 

percent of positive responses to questions on D4, non-punitive response to error. 

Across CYs 2010 to 2013, SE reporting rates were calculated per 1,000 dispositions (hospital 

discharges) for each of the Services. The Army SE reported rate was 0.223, the Navy rate was 

0.375, Air Force rate was 0.539, and the NCR MD (which began reporting in December 2012) 

had a rate of 0.291 for its reporting period. No distinctions were made between SEs in 

ambulatory settings and inpatient facilities.

Tables 5.9 and 5.10 demonstrate the top five SE categories across the direct care component by 

fiscal year and Service. The individual Services and yearly distributions varied slightly in the 

most common SE categories but the common top three categories across all Services were: 

retained foreign object, unanticipated death-adult, and wrong site surgery. Notably, delay in 

treatment was among the top five SE categories for the Air Force only. 

Table 5.9 Top 5 Sentinel Events by Year

2010 2011 2012 2013

1 Unanticipated 

Death-Adult 19 Retained Foreign Object 21 Unanticipated DeathAdult 18 Retained Foreign Object 17 

2 Retained Foreign 

Object 17 Wrong Site Surgery 13 Retained Foreign Object 16 Unanticipated Death-Adult 13 

3 Wrong Site Surgery 10 Unanticipated 

Death - Infant 7 Unanticipated Death - Infant 11 Wrong Site Surgery 11 

4 Unanticipated Death 

- Infant 9 Unanticipated Death-Adult 7 Loss of Function 10 Delayed Treatment 10 

5 Loss of Function 8 Delayed 

Treatment 6 Delayed Treatment 9 Procedural Complication 10 

2014 MHS Review Group 

Source: Patient Safety Reporting System, DoD Patient Safety Analysis Center (PSAC), June 2014

165
August 29, 2014 5. Patient Safety in the Military Health System

Table 5.10 Top Five Sentinel Events by Service with Frequency Count, 2010 – 2013

 DoD 

Overall Air Force Army Navy NCR MD

1 

Retained 

Foreign 

Object

71 Delayed 

Treatment 15 Retained Foreign Object 34 Retained Foreign Object 23 Suicide 3 

2 Unanticipated 

Death-Adult 57 Retained Foreign Object 13 Unanticipated Death-Adult 25 Unanticipated Death-Adult 18 Unanticipated Death-Adult 2 

3 Wrong Site 

Surgery 40 Unanticipated Death-Adult 12 Wrong Site Surgery 19 Unanticipated Death-Infant 16 

4 Unanticipated 

Death – Infant 34 Wrong Site Surgery 10 Unanticipated Death-Infant 11 Loss of Function 13 

NCR MD 

Reported 1 each 

in all of the 

remaining 9 SE 

categories

5 Delayed 

Treatment 28 Medication Error 6 Loss of Function 10 Wrong Site Surgery 10 

2014 MHS Review Group 

Source: Patient Safety Reporting System, DoD Patient Safety Analysis Center (PSAC), June 2014

External Health System Comparison 

Frequency of SE reports were compared to the MTFs using data from two systems that provided 

SE information. Health System 2 provided denominator data in discharge days allowing SE 

rates to be calculated, assuming that 100 percent of SEs were accounted for (versus only reported 

SE). 

Civilian Health Systems Data: Health System 2 provided counts of SEs and discharge days 

(denominator) for SEs by quarter from Q1 2010 to Q4 2013 (4 calendar years of data). With 

numerator and denominator data, SE rates were calculated. However, detail on the types of SEs 

that were reported was not provided. Health System 3 provided counts of SE reports by SE type 

and by level of harm (level of harm reported in RCA comparison section) by quarter from Q1 

2010 to Q4 2013 (4 calendar years of data). Discharge Days information was not provided. 

Direct care SE data were available from FY 2010 to FY 2013. Due to differences in FY vs CY, 

Health Systems 2 and 3 data had to be aggregated at the FY level for comparisons (see Figures 

5.5 and 5.6). 

External Health System Comparison Limitations: The direct care rate of SEs was calculated 

using all reported SEs in FY 2011 to FY 2013 as numerator and hospital discharge days as the 

denominator; however, no distinction was made between SEs in ambulatory settings and 

inpatient facilities. The underlying assumption in calculating SE rates is that these occurred in 

hospitals. Additionally, to make valid comparisons, both systems should use the same definition 

of SE’s. Health System 3 uses additional SE types beyond those used in the direct care 

component. 

166 
Military Health System Review – Final Report August 29, 2014

Figure 5.5 Number of SEs across Direct Care, Health System 2, and System 3, FY11 – FY13 

2014 MHS Review Group

Source: DoD Patient Safety Reporting System, TRICARE Management Activity (TMA)/Health Affairs (HA), July 2014

Figure 5.6 SE Rates per 1,000 Discharges, Direct Care and Health System 2, FY11 – FY13

2014 MHS Review Group

Source: DoD Patient Safety Reporting System, TRICARE Management Activity (TMA)/Health Affairs (HA), July 2014

0

10

20

30

40

50

60

70

80

90

100

FY11 FY12 FY13

# of Sentinel Events

Fiscal Year

DoD

HS2

HS3

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

FY11 FY12 FY13

SE Rates p 1,000 discharges

Fiscal Year

DoD

HS2

167
August 29, 2014 5. Patient Safety in the Military Health System

External Health System Comparison Results: Over three fiscal years, the direct care 

component reported a total of 257 SEs, Health System 2 had 65 SEs, and Health System 3 had 

171 SEs. However, rates are more appropriate for comparison as they adjust for differences in 

population size (discharge days, bed days). When comparing rates of SE across FYs for Health 

System 2 and direct care, direct care reported half the rate of SEs in comparison with Health 

System 2 for FY 2011 (0.282 per 1,000 discharges vs 0.667 per 1,000 respectively). 

Findings Regarding Sentinel Events

In comparison to another system, there is reason to believe the direct care component performs 

similarly to civilian health care systems, and may actually perform better. However, this was 

just one system with caveats that have to be considered with regard to the data analysis. 

1. DoD’s SE definition matches that of The Joint Commission, but does not provide 

sufficient clarity for consistent decision making because of local interpretation. 

2. Systematic progress to decrease the overall trend regarding number and type of 

occurrences within any SE category is not evident. 

 Recommendations Regarding Sentinel Events (SE) Data

a. Clarify policy and educate health care staff on the SE definition and event types to 

reduce variation in interpretation. 

b. MHS governance should pursue an enterprise-wide improvement process 

addressing the top five reported SEs, improve the distinction between ambulatory 

versus hospital settings, and monitor SE occurrence by rates using appropriate 

denominator estimates. 

Root Cause Analysis 

RCA is a systematic approach to determining the true root cause of an event or accident and 

separating the root cause(s) from other contributing factors, with the goal of preventing events or 

accidents from recurring. An RCA is required by DoDM 6025.13 for all SEs (see definition in 

Measure 4 above). Per DoDI 6025.13, TJC reviewable SEs must also be reported to TJC if the 

facility is accredited by TJC. The Accreditation Association for Ambulatory Heath Care 

(AAAHC) requires review of adverse events at the time of accreditation. Per DoD policy, an 

RCA investigation must be completed by the MTFs on all SEs, including TJC-reviewable SEs 

within 45 calendar days of the MTF becoming aware of the SE (see Appendix 5.9 for list of TJC 

defined reviewable SEs). 

All SEs/adverse events must be reported to DHA. Corresponding RCAs are forwarded to the 

DoD Patient Safety Analysis Center (PSAC). However, there is no DoD policy requiring that 

RCAs be completed for non-SEs nor be submitted to the PSAC. In addition, per individual 

Service policies, RCAs may be required on incidences not meeting the SE definition; however, 

these RCAs need not be forwarded to PSAC. 

168
Military Health System Review – Final Report August 29, 2014

There is no established process for communicating RCA feedback to staff or the PSAC. RCA 

corrective actions and follow up of completed events need not be reported to DoD. There is no 

process to cross reference a single event within the current systems (Patient Safety Reports, 

Centralized Credentialing and Quality Assurance System)

65. 

Analysis

The purpose of this analysis is to account for all RCA investigations completed by the Services 

and NCR MD at the MTFs. RCA investigations are characterized by event type, date, and 

harm/outcome to determine emerging trends over time. 

Table 5.11 shows the number of RCAs by Service, by year. 

Table 5.11 Number of RCAs reported to PSAC, DHA, and Health Affairs by FY of Event Date

FY 2010 FY 2011 FY 2012 FY 2013 Total

DoD 105 84 114 85 388

Air Force 28 21 35 23 107

Army 45 36 49 31 161

Navy 32 27 30 25 114

NCR MD N/A66 N/A N/A 6 6

2014 MHS Review Group

Source: Patient Safety Reporting System Database, June 2014

65 The Centralized Credentials Quality Assurance System is a Web-based, worldwide credentialing, 

privileging, risk management and adverse actions database for the Defense Health Agency.

66 N/A: The NCR MD was established in December 2012.

169

 
August 29, 2014 5. Patient Safety in the Military Health System

Table 5.12 shows the number of RCAs by event type for all Services for the period of review. 

Table 5.12 RCAs by Event Type submitted to PSAC, FY10 – FY13 (rank ordered)

Type Number Reported

Unanticipated Death (all ages) 110

Surgery on Wrong Patient or Body Part 74

Foreign Body, Unintended Retention 71

Loss of Function, Major Permanent 47

Non- TJC Reviewable 38

Suicide, 24 Hour Care/within 72 hours of Discharge 18

No Type Provided/Blank 16

Radiation Overdose 4

Medical 3

Surgical 3

Neonatal Hyperbilirubinemia, Severe 2

Rape 1

Infant Discharged to Wrong Family 1

2014 MHS Review Group

Source: Patient Safety Reporting System Database, June 2014

170
Military Health System Review – Final Report August 29, 2014

Of the 388 RCA reports submitted to PSAC, the top three categories were Unanticipated Death, 

Wrong Site Surgery, and Retained Foreign Object. Figures 5.7 through 5.9 display four event 

types by Service and non-JCAHO (JCAHO is the former name of TJC) categorized events 

submitted to PSAC during FYs 2010 to 2013.

Figure 5.7 Air Force Top 4 Event Types for RCA Reports Submitted, FY10 – FY13 

2014 MHS Review Group

Source: RCA: Patient Safety Reporting System Database, June 2014

11

8

11

7

5

4

6 6

3

4

6

4

3

4

5

2 2

1

3

1

0

2

4

6

8

10

12

2010 2011 2012 2013

# of RCA Event Types 

Fiscal Year 

Death, unanticipated,

any age

Foreign body,

unintended retention

Surgery on wrong

patient or body part

Non-TJC event

Suicide, 24-hour care /

within 72 hrs of

discharge

171
August 29, 2014 5. Patient Safety in the Military Health System

Figure 5.8 Army Top 4 Event Types for RCA Reports Submitted, FY10 – FY13

2014 MHS Review Group

Source: RCA: Patient Safety Reporting System Database, June 2014

Figure 5.9 Navy Top 4 Event Types for RCA Reports Submitted, FY10 – FY13

2014 MHS Review Group

Source: Self-reported by Service to the Patient Safety Program, June 2014 (Navy)

9

10

13

9

7 7

4 4

3

2

4 4

3

2 2

3

8

4 4

5

0

2

4

6

8

10

12

14

2010 2011 2012 2013

# of RCA Event Types 

Fiscal Year 

Death, unanticipated,

any age

Foreign body,

unintended retention

Surgery on wrong

patient or body part

Non-TJC event

Loss of function, major

permanent

14

5

13

10

5

11

10

9

19

11

4

6

0 0

1

2 2 2

12

4

0

2

4

6

8

10

12

14

16

18

20

2010 2011 2012 2013

# of RCA Event Types 

Fiscal Year 

Death, unanticipated,

any age

Foreign body,

unintended retention

Surgery on wrong

patient or body part

Non-TJC event

Loss of function, major

permanent

172
Military Health System Review – Final Report August 29, 2014

Table 5.13 describes the level of harm results for RCA investigations by Service and year for 

FYs 2010 to 2013.

Table 5.13 Level of Harm Results for RCA Investigations by FY and Service, FY10 – FY13

Fiscal 

Year Death

Permanent 

loss of 

function

No loss of 

function Undeterminable (blank) NR

Grand 

Total

Air Force

2010 14 3 8 3 28

2011 6 12 3 21

2012 11 2 17 2 3 35

2013 7 4 8 2 2 23

Army

2010 15 2 28 45

2011 7 2 25 2 36

2012 11 7 6 25 49

2013 10 5 11 2 3 31

Navy

2010 13 9 8 2 32

2011 12 4 10 1 27

2012 16 3 8 3 30

2013 10 6 6 3 25

NCR MD

2013 2 4 6

Total 134 47 151 45 8 3 388

2014 MHS Review Group

Source: Patient Safety Reporting System Database, June 2014

External Health System Comparison Methods

Health System 3 provided detailed RCA data for SEs containing level of harm results for FYs 

2011 to 2013. These results were compared to direct care RCA level of harm results for the 

same time period. 

External Health System Comparison Limitations: There is no means of one-to-one 

comparisons based on frequency of SE events alone. Health System 3’s SE reporting categories 

are incompletely defined and include additional SE types beyond TJC categories. Additionally, 

Health System 3’s requirement for conducting RCAs is unknown.

External Health System Comparison Analysis: Over three fiscal years, the direct care 

component reported a total of 240 level of harm results for SE only RCAs where there was a 

173
August 29, 2014 5. Patient Safety in the Military Health System

level of harm reported (see Table 5.14), while Health System 3 had 171 level of harm results for 

SE only RCAs (see Table 5.15). 

The two most frequently occurring level of harm results for the direct care component were 

“death” and “no loss of function” across all three years. On average, death occurred 37 percent 

of the time for SE RCAs with reported outcomes. 

The two most frequently occurring level of harm results for System 3 were “no harm” and 

“death”, respectively. On average, death occurred 25 percent of the time for SE RCAS across 

three fiscal years within Health System 3. 

“No harm” and “no loss of function” are not comparable categories across the direct care 

component and Health System 3. The only comparable level of harm outcome is death, which is 

more commonly reported for SE RCAs in direct care than for Health System 3. However, rates 

are preferable to frequency of events when comparing across systems because the underlying 

population differences are mitigated with rate comparisons. 

Table 5.14 Direct Care SE RCA, Level of Harm Findings, FY11 – FY13

Level of Harm FY 2011 FY 2012 FY 2013 Total

n % n % n %

Death 24 33% 37 39% 28 38% 89

No loss of function 41 56% 20 21% 26 36% 87

Permanent loss of function 6 8% 12 13% 14 19% 32

Undeterminable 1 1% 24 26% 2 3% 27

Missing (blank) 1 1% 1 1% 1 1% 3

Not Reported 0% 0% 2 3% 2

Total 73 100% 94 100% 73 100% 240

2014 MHS Review Group

Source: Patient Safety Reporting System Database, June 2014

174
Military Health System Review – Final Report August 29, 2014

Table 5.15 System 3 SE RCA, Level of Harm Findings, FY11 – FY13

Level of Harm FY 2011 FY 2012 FY 2013 Total

n % n % n %

No Harm 14 23% 18 28% 12 26% 44

Death 27 44% 8 13% 8 17% 43

Moderate 7 11% 26 41% 6 13% 39

Major-Temporary 6 10% 7 11% 10 22% 23

Minor 4 7% 3 5% 8 17% 15

Major-Permanent 2 3% 1 2% 1 2% 4

Emotional Injury Only 1 2% 1 2% 1 2% 3

Total 61 100% 64 100% 46 100% 171

2014 MHS Review Group

Source: Patient Safety Reporting System Database, and External Health System 3 Data, June 2014

Findings Regarding Root Cause Analysis (RCA)

1. Based on historical RCA analysis and current data, the content of RCAs remains highly 

variable across all Services and event types. RCAs associated with the most serious 

events often provide very limited insight into the factors that may be corrected to prevent 

recurrence. RCAs should be reviewed not as a requirement but for learning and system 

improvements. Based on historical RCA PSAC analyses, no consistent follow-up 

process exists to assess process improvement following an RCA. Across the Services 

and at the MTF level, information gleaned from completed RCAs is not widely shared for 

frontline staff to make improvements where possible. Lack of a common identifier for 

events does not allow for cross-referencing or follow up of events once an RCA is 

completed. 

 Recommendations Regarding Root Cause Analysis (RCA)

a. Establish clear expectations for the RCA process and the follow up that will occur.

Performance Improvement Root Cause Analysis 

In June 2014, each Service (Army, Navy, Air Force, and NCR MD) provided a list of all RCAs 

that were conducted for performance improvement purposes. These RCAs were performed for 

events that did not meet SE criteria. 

“Performance Improvement” (PI) RCA is a term agreed on by the MHS Review Group to 

describe RCA investigations conducted to identify variation in performance, systems, and 

processes; to train or remain current on RCA competency; and for use in Probability Risk 

Assessments. The RCA information is maintained at the Service and MTF levels. These data 

include all PI RCAs between FY 2010 and FY 2013 reported by the Services to the MHS 

Review Group for the purposes of this review (NCR MD data only include December 2012 to 

December 2013). The Services were asked to provide: Service, year of event, MTF name, event 

175
August 29, 2014 5. Patient Safety in the Military Health System

type, level of harm, and to state whether the RCA was conducted for training purposes. A total 

of 425 PI RCAs were reported to the MHS Review Group. Eighty-one of the Navy (102 total) 

and 7 of the Army (174 total) PI RCAs were identified as RCAs conducted for training purposes 

or proactive risk reviews. 

Table 5.16 shows the Services’ different methods for classifying event type and reporting their 

RCA events and the total number of PI RCAs submitted. 

Table 5.16 Service Identified Source for RCA Classification of Event Type and Total Number of 

PI RCAs

Service RCA Classification Number of PI RCAs Reported

Air Force PSR Categories 131 events

Army Not Specified 174 events (two events had no specified date)

Navy DoD Short Form 102 events

NCR MD DoDM 6025.13 guidance 18 events

2014 MHS Review Group

Source: Self-reported by Services to the Patient Safety Program, June 2014 (Navy), June 2014 (Air Force), July 2014 

(Army), and July 2014 (NCR-MD)

Figure 5.10 shows the Services PI RCAs by calendar year. This figure demonstrates an 

increased number of PI RCAs across direct care each year, over the last four years. 

Figure 5.10 PI Service RCAs by Year and Service

2014 MHS Review Group

Source: Self-reported by Services to the Patient Safety Program, June 2014 (Navy), June 2014 (Air Force), July 2014 

(Army), and July 2014 (NCR-MD)

2010 2011 2012 2013

NCRMD 0 0 5 13

Navy 27 24 23 28

Army 35 41 46 50

Air Force 30 29 41 31

0

20

40

60

80

100

120

140

# of PI RCAs

176
Military Health System Review – Final Report August 29, 2014

Table 5.17 demonstrates the top PI RCAs reported by each Service and the NCR MD for the last 

four consecutive years. The direct care data include the combined data sets of all the Services’ 

PI RCAs. These were consolidated into TJC RCA event types. Overall, suicide was the largest 

event category with a total of 156 events. 

Table 5.17 Top PI RCAs for DoD Overall, Air Force, Navy, Army, and NCR MD

Service Top PI RCAs Number of PI RCAs

1. Suicide 156

2. Other Unanticipated Events 82

DoD Overall (TJC Classification) 3. Delay in Treatment 60

4. Medication Error 56

5. Med-Equipment related 18

1. Suicide 52

2. Delay in Diagnosis/Treatment 23

Air Force (TapRooT Software Classification) 3. Medication/IV fluid/biological 14

4. Clinical Process or Procedures 12

5. Unanticipated Death 8

1. Delay in Diagnosis/Treatment 34

2. Medication-related Event 15

Navy (DoD Short Form Classification) 3. Other 14

4. OB Related: Other 11

5. Patient Suicide/Risk of 7

1. Suicide 67

Army* (Classification not specified) 2. Other 5

3. Blank 4

NCR MD (DoDM 6025.13) 1. Medication Error 4

2. Suicide Gestures 2

*Army had 67 unstandardized types 

2014 MHS Review Group

Source: Self-reported by Services to the Patient Safety Program, June 2014 (Navy), June 2014 (Air Force), July 2014 

(Army), and July 2014 (NCR-MD)

Findings Regarding Root Cause Analysis for Performance Improvement

1. In addition to RCA associated with reviewable sentinel events, MTFs exceeded policy 

DoDM 6025.13 by conducting 425 RCAs for performance improvement purposes in an 

effort to identify and correct systemic process issues. 

2. Variations are found in RCA event type classifications, demonstrating an overall lack of 

consistent categorization. Not all Services forward PI RCAs to the PSAC, so there is no

complete database to learn from and establish safe practices.

177
August 29, 2014 5. Patient Safety in the Military Health System

 Recommendation Regarding Root Cause Analysis for Performance Improvement

a. Standardize the PI RCA process with a focus on event type classifications, a 

centralized repository, and dissemination of the lessons learned. 

Patient Safety Reporting System (PSRS)

The PSRS was fully implemented enterprise-wide in June 2011. Therefore, complete patient 

safety reporting data are available only for FY 2012 and FY 2013. The PSRS is a web-based, 

self-reported, anonymous, commercial off-the-shelf reporting application that consolidates both 

medication and non-medication reporting using a standardized taxonomy to improve 

aggregation, trending, and analysis. Use of the PSRS was voluntary but highly encouraged as a 

reporting system between June 2011 and October 2013. In October 2013, patient safety 

reporting became mandatory with the publication of the current DoDM 6025.13. 

PSRS events are categorized by harm categories, including the following: 

1. Near Miss: did not reach the patient and unsafe condition 

2. No-Harm: no harm to the patient and emotional distress 

3. Harm: additional treatment, temporary harm, permanent harm, severe permanent harm, 

and death

Figure 5.11 shows the increase in patient safety reporting by month between FY 2012 and FY 

2013.

Figure 5.11 Total PSR Events by Month, FY12 – FY13

2014 MHS Review Group

Source: PSR: Patient Safety Reporting System Database, April 2014

0

2000

4000

6000

8000

10000

12000

14000

16000

Number of Events

Fiscal Year

Total Events Linear (Total Events)

178
Military Health System Review – Final Report August 29, 2014

Although this trend is desirable, when compared to 2011 HSOPS data,67 there has been little 

progress in increasing the number of staff who report at least one event over a 12-month period. 

In 2011, only 27 percent of staff completing the HSOPS responded positively to this question. 

This puts DoD within the 10th percentile (underperforming) for patient safety reporting when 

compared to the AHRQ HSOPS national average. According to the Institute for Healthcare 

Improvement (IHI) Global Trigger Tool for Measuring Adverse Events (see Appendix 5.4), 

written in 2009, “voluntary reporting approaches can be subjective and unless events are 

particularly salient patient safety issues maybe underreported by as much as 80-90%.” 68 For 

these reasons, the IHI does not recommend the use of self-reporting systems to determine harm 

rates.

Reported Near Miss and No Harm events show an increasing trend over time. Among the 

Services there is significant variance in Near Miss reporting with Army reporting an average 

1,566 events per month, Air Force 1,109, and Navy 428. Army is averaging 1,290 No Harm 

event reports monthly with Air Force at 748 and Navy at 615. The overall trend in reported 

Harm events for the Services has remained relatively flat over the past two fiscal years with

Army reporting an average of 270 per month, Navy 166 per month and Air Force 101 per month

(Figure 5.12).

Figure 5.12 Events by Harm by Month, FY12 – FY13

2014 MHS Review Group

Source: PSR: Patient Safety Reporting System Database, retrieved April 2014

67 AHRQ. Hospital Survey of Patient Safety Culture: 2011 User Comparative Database Report. Available at: 

http://www.ahrq.gov/legacy/qual/hospsurvey11/ 68 Classen, DC, et al. Global trigger tool’ shows that adverse events in hospitals may be ten times greater than 

previously measured. Health Affairs 2011 Apr;30(4):581-9. doi: 10.1377/hlthaff.2011.0190.

0

500

1000

1500

2000

2500

3000

3500

4000

Number of Events

Fiscal Year 

Near Miss NO HARM HARM

Linear (Near Miss) Linear (NO HARM) Linear (HARM)

179

 
August 29, 2014 5. Patient Safety in the Military Health System

Self-reporting tools like PSR are used internationally to capture, aggregate, and trend untoward 

medical event data. James Reason, PhD, risk analysis and accident causation expert, suggests, 

“A reporting culture means cultivating an atmosphere where people have confidence to report 

safety concerns without fear of blame. Employees must know that confidentiality will be 

maintained and that the information they submit will be acted upon, otherwise they will decide 

that there is no benefit in their reporting...Leadership is central to safety culture.”69 Results 

from HSOPS and site visit observations (discussed later in this report) such as fear of retribution 

or punitive environment may influence the likelihood of staff reporting events using the PSR

tool. 

Findings Regarding the Patient Safety Reporting System

1. There are inconsistent event reporting processes (identification of events, staff reporting 

of events, approval of events, and classification of events) across all Services and MTFs.

2. Less than 30 percent of staff actively participates in reporting patient safety events 

according to the most recent culture survey, with no changes observed over time. DoD 

results fall at the 10th percentile for reporting when compared to the civilian benchmark. 

Based on HSOPS data, there have been no improvements in the number of staff who have 

reported at least one event over a 12-month time period. 

3. The PSRS does not provide an accurate indication of the system’s harm level or harm 

rate. 

 Recommendations Regarding the Patient Safety Reporting System

a. Standardize the event type components of the event reporting process. 

b. Standardize leadership activities to drive a culture of safety (i.e., Executive Toolkit). 

c. Adopt a chart audit based methodology such as the IHI Global Trigger Tool (GTT) 

to determine harm rate. 

Measures within Purchased Care Settings

As set forth in the TRICARE Operations Manual (TOM), Chapter 7, Section 4, the contractors 

are required to use the most current NQF Serious Reportable Events (SREs) and AHRQ PSIs as 

a mechanism to identify, track, trend, and report interventions to resolve potential quality issues 

(QIs) and confirmed quality issues. 70 Additionally, the contractor must report potential SREs to 

the TRICARE Regional Office (TRO) or TRICARE Area Office (TAO) or Designated Provider 

Program Office (DPPO) within two business days from when the contractor becomes aware of 

69 Reason, J. (1997). Managing the risks of organizational accidents. Aldershot: Ashgate.

70 A potential quality issue (QI) is defined as a clinical or system variance warranting further review and 

investigation for determination of the presence of an actual QI. A confirmed QI is defined as a verified deviation, as 

determined by a qualified reviewer, from an acceptable standard of practice or standard of care as a result of some 

process, individual, or institutional component of the health care system. 

180

 
Military Health System Review – Final Report August 29, 2014

the event and closure of the reported SRE is required within two business days to include 

summary of actions taken. Each contractor uses a mix of standardized reporting matrices as well 

as individual best practice matrices to monitor and report patient safety concerns. The 

TRO/TAO or DPPO office provides oversight for their respective contractor processes and 

compliance of the requirements in accreditation, clinical credentialing, and clinical 

quality/patient safety. 

All of the regional contractors have processes in place to review patient safety and quality of care 

issues. The contractor must assess every medical record reviewed for any purpose and any care 

managed/observed/monitored on an ongoing basis for PQIs. The contractor is further directed to 

implement appropriate quality interventions using evidence-based medicine/guidelines and best 

medical practices to reduce the number of QIs and improve patient safety. When the contractor 

confirms a QI, the determination should include assignment of an appropriate severity level 

and/or sentinel event, and describe the actions taken to resolve the quality problem. 

Reporting of patient safety, patient harms, or quality-of-care issues is voluntary for civilian 

providers. Contractors have developed various sources in attempting to identify issues in 

addition to claims data; for example, beneficiary complaints, MTF concerns for enrolled 

beneficiaries, governmental inquiries, concurrent review processes for inpatient admissions, and 

medical records from focus studies. In presenting the aggregate data from the contractors, every 

effort was made to translate the heterogeneous mixtures of mandatory reporting metrics and 

additional best practice metrics from multiple disparate sources into homogenous measures to 

facilitate comparison; however, direct comparisons remain challenging.

Agency for Healthcare Research and Quality Patient Safety Indicators (PSIs)

The AHRQ PSI set is a useful screening tool for highlighting areas in which quality should be 

further investigated by hospitals and for oversight in health plans. AHRQ PSIs also provide a 

useful benchmark for facilities in tracking progress in quality improvement. The AHRQ PSIs 

were designed for providers of care, not for health plans; however, these indicators are used as a 

proxy measure for TRICARE to identify potential quality of care issues. Contractors are 

directed through the TOM to use current PSI software to evaluate the safety of care delivered in 

the network. The contractor is required to analyze the results to identify PQIs and patient safety 

issues for individual providers, groups, and/or facilities. An official analysis must be provided in 

their required Clinical Quality Management Program Annual Report. 

The AHRQ PSIs are homogenous and comparable among the contractors, as they all use the 

AHRQ standardized methodology from claims data. The data can be compared against the 

national average benchmarks published by AHRQ.

Methodology/Benchmark or National Comparison Information: The TRICARE data 

presented in this document are shown with AHRQ-generated nationwide comparative rates for

the AHRQ QITM PSIs. The AHRQ comparison rates are based on analysis of 44 States from 

AHRQ’s 2010 Healthcare Cost and Utilization Project (HCUP) State Inpatient Databases. The 

QI observed rate for provider-level indicators is scaled to a rate per 1,000 persons at risk. 

181
August 29, 2014 5. Patient Safety in the Military Health System

TRICARE PSI rates indicate risks or harms that may have been encountered by MHS 

beneficiaries while hospitalized in purchased care facilities. It is important to note that 

TRICARE is only able to capture incidence of risk or harms across multiple facilities. Currently 

there is only one available AHRQ-specific stratification/benchmark for commercial, Medicare, 

Medicaid and “other” payers to characterize risks or harms—in other words, no such 

stratification/benchmark exists for TRICARE. 

TRICARE data were obtained from each continental United States (CONUS) region for the most 

recent four fiscal years (October 2010 – September 2013) and 18 PSI measures were analyzed: 

PSI 2 through PSI 19. Overall, the majority of measures were below the national average and a 

few were above the national average (see Table 5.18). Data from outside continental United 

States (OCONUS) and Designated Providers showed overall small numbers of events with 

differences in reporting methodology, which made aggregation for analysis, challenging. 

Table 5.18 PSI Rates for Purchased Care Regions Compared to AHRQ National Benchmarks, FY 

10 – FY13

Lower than Benchmark

Higher than Benchmark

National Benchmark

2014 MHS Review Group

Source: Data from Annual Reports from United Healthcare Military and Veteran, July 2014

Potential Quality Issues (PQIs) and Quality Issues (QIs)

The overall number of PQIs identified varied among the contractors but a greater difference was 

observed in the confirmed quality findings. The contractors were compared according to the 

AHRQ PSIs, SREs, and Hospital Acquired Condition (as defined by CMS for claims coding 

methodology for DRG payment), as these were homogenous comparable indicators among 

contractors. The other indicators were specific to the various contractors and were not 

comparable. The data demonstrate that the contractors’ processes were effective in identifying 

patient care quality and safety issues despite facility and provider voluntary reporting. There are 

no national or other benchmarks available for comparison (see Figures 5.13 and Figure 5.14)

*Nat *Nat *Nat

Average Average Average

Death in Low Mortality DRGs (PSI 2) 0.14 0.01 0.27 0.01 0.00 0.12 0.03 0.10 0.00 0.59 0.03 0.12 0.07 0.59 0.13

Decubitis Ulcer (PSI 3) 1.67 0.22 0.18 0.40 1.52 0.30 2.00 0.41 4.10 0.18 2.23 5.18 4.11 1.19 2.23 7.27

Failure to Rescure (PSI 4) 0.00 6.57 118.57 7.68 83.07 12.40 36.20 6.36 77.94 12.71 94.27 77.51 95.36

Foreign Body Left During Procedure (PS 5) 0.05 0.06

Iatrogenic Pneumothorax (PSI 6) 0.20 0.11 0.13 0.38 0.21 0.16 0.07 0.43 0.50 0.19 0.91 0.43 0.45 0.20 0.91 0.14

Selected Infections Due to Medical Care (PSI 7) 0.38 0.16 0.45 0.43 0.25 0.17 0.44 0.39 0.18 1.13 0.75 0.59 0.32 1.13 0.67

Postoperative Hip Fracture (PSI 8) 0.00 0.00 0.03 0.00 0.00 0.03 0.00 0.00 0.00 0.03 0.00 0.06

**Perioperative Hemorrhage or Hematoma (PSI 9) 3.72 4.35 1.39 5.86 1.70 1.98 1.42 2.42 1.30 1.95 3.87 2.37 2.34 1.80 3.87 2.21

Postoperative Physiologic and Metabolic Derangement (PSI 10) 0.43 0.16 0.50 0.59 0.44 0.52 0.10 0.15 0.00 0.48 0.19 0.00 0.55

Postoperative Respiratory Failure (PSI 11) 5.44 5.07 3.81 8.61 5.49 6.27 2.85 8.59 5.00 4.23 2.32 8.17 6.00 3.55 2.47 5.79

Postoperative PE or DVT (PSI 12) 6.56 2.43 2.11 4.51 6.47 2.29 2.12 6.25 3.90 2.51 2.54 7.28 7.37 2.26 2.45 7.10

Postoperative Sepsis (PSI 13) 6.88 3.51 6.38 12.00 10.80 8.72 5.42 10.72 6.60 6.37 3.05 10.74 6.86 4.05 3.05 10.51

Postoperative Wound Dehiscence (PSI 14) 0.68 0.80 1.38 1.85 0.87 0.24 0.42 2.02 1.90 0.65 1.45 2.17 1.91 0.60 1.45 1.53

Accidental Puncture or Laceration (PSI 15) 2.41 2.63 3.13 2.45 2.55 2.70 3.81 2.68 3.20 3.34 5.19 2.88 4.02 3.31 5.48 3.93

Transfusion Reaction (PSI 16) 0.00 0.00 0.00

Birth Trauma Rate - Injury to Neonate (PSI 17) 2.06 1.82 2.10 2.39 2.50 2.15 2.73 2.15

Obstetric Trauma - Vaginal Delivery with Instrument (PSI 18) 145.30 132.74 152.84 139.11 138.46 126.73 129.82 143.03 137.20 136.68 168.89 146.39 145.37 146.64 161.48 167.98

Obstetric Trauma - Vaginal Delivery without Instrument (PSI 19) 21.46 23.51 24.23 22.46 20.05 24.76 19.74 22.14 21.70 21.97 28.68 23.78 22.93 26.71 29.53 28.84

FY 2013 FY 2012 FY 2011 FY 2010

North 

Region

South 

Region 

West 

Region

*Nat 

Average

North 

Region

West 

Region

North 

Region

South 

Region 

West 

Region

Agency for Healthcare Research and Quality (AHRQ) Patient 

Safety Indicators (PSIs) 

North 

Region

South 

Region 

West 

Region

South 

Region 

182
Military Health System Review – Final Report August 29, 2014

Figure 5.13 Total Number of Quality Issues (QIs) for AHRQ PSIs, HACs, SREs Identified in FY10 –

FY13 for Purchased Care

2014 MHS Review Group

Source: Managed Care Support Contractors Annual Report, June 2014

Reviewing all three CONUS TRICARE contractors in aggregate over the past four years shows 

an increase in total PQIs identified in FY 2011 and decreasing numbers in FY 2012 and FY 

2013. In evaluating the individual regions, the West has generally reported higher levels of 

AHRQ PSIs, HACs, and SREs compared to the other two regions with initially what appeared to 

be a significant spike in FY 2012 that appeared to cluster in the area of obstetrical/newborn 

issues. Further research into this data revealed a combination of neonatal trauma and obstetrical 

trauma into the reporting category of “birth trauma.” When this was corrected to “birth trauma 

injury to neonate” the data fell within the expected statistical range, and this latter point was used 

in the graphical representation.

FY2010 FY2011 FY2012 FY2013

Number of PQIs 2280 3591 3108 1343

0

500

1000

1500

2000

2500

3000

3500

4000

Number of PQIs

183
August 29, 2014 5. Patient Safety in the Military Health System

Figure 5.14 Total Number of Quality Issues (QIs) for AHRQ PSIs, HACs, SREs Identified in FY10 –

FY13, by Region for Purchased Care 

2014 MHS Review Group

Source: Managed Care Support Contractors Annual Report, June 2014 

HAC = CMS defined Hospital Acquired Conditions not present on admission

All contractors count cases by self-selected PQI/QI case attributes that may include: in each case 

investigated, multiple indicators or issues that may be identified in the case, and/or by number of 

involved providers which may be evaluated in the given segment of care. The methodology used 

to identify number of cases worked reflects contractor-unique practices that make comparison of 

potential quality issues and/or actual quality issues difficult.

Serious Reportable Events

The contractors are required to use the most current NQF SRE indicators as a source for potential 

serious quality of care issues. There is no mandatory reporting for civilian facilities and 

providers, although the contractors have developed processes for identification. 

FY2010 FY2011 FY2012 FY2013

North 104 149 72 39

South 53 70 21 43

West 156 126 130 144

0

20

40

60

80

100

120

140

160

180

Number of QIs

North

South

West

184
Military Health System Review – Final Report August 29, 2014

Figure 5.15 Total Number of National Quality Forum Serious Reportable Events in FY10 – FY13,

by Region for Purchased Care 

2014 MHS Review Group

Source: Managed Care Support Contractors Annual Report, June 2014

In examining the three individual region numbers, the only notable data outlier is in 2012 in the 

West region where there was a significantly higher number of SREs in comparison to the North 

and South regions. Further detail reveals the majority of this spike is accounted for by 23 patient 

falls that were reviewed and assigned a Severity Level 1, meaning that a QI was present with 

minimal potential for significant adverse effects on the patient. There are no benchmarks 

available (see Figure 5.15).

A high-level impression of the purchased care data in aggregate for the past four years is that 

overall rates for the majority of tracked metrics are at or below the national averages. It is 

important to understand that comparison of purchased care data with direct care data is 

problematic. Reporting of the indicators to the TRICARE contractors that administer benefits 

and pay claims in the purchased care component is voluntary, unlike in direct care where 

reporting is mandatory. The majority of possible safety and quality concerns arise through 

claims review, beneficiary complaints, record reviews and other active monitoring sources and 

processes. Thus, comparing voluntary civilian rates to a system with mandatory reporting may 

inappropriately give the appearance that the direct care component has higher rates of adverse 

safety issues. 

Gaps and Findings Regarding Patient Safety in Purchased Care

The major gap in identifying patient harm and other potential safety issues for the TRICARE 

population treated by civilian providers and facilities is the voluntary reporting process. The 

only mechanism for mandatory reporting of patient harm/safety issues for TRICARE would be 

FY2010 FY2011 FY2012 FY2013

North 10 13 6 5

South 9 5 9 7

West 20 11 35 19

0

5

10

15

20

25

30

35

40

Number of Events

North

South

West

185
August 29, 2014 5. Patient Safety in the Military Health System

through a congressional action tying reporting to claims payment. The current DHA/contracting 

reimbursement methodology does not provide the framework for flexibility in reimbursement 

rates negotiation by a contractor.

1. For the past four years, overall rates for the majority of tracked patient safety metrics are 

at or outperformed national benchmarks. Review of aggregate data for the three CONUS 

contractors over the past four years shows an increase in total PQIs identified in FY 2011 

(unknown if due to increased events or increased reporting) and then steadily decreasing 

numbers in FY 2012 and FY 2013. 

2. In evaluating the individual regions, the West has generally reported higher levels of 

AHRQ PSIs, HACs, and SREs compared to the other two regions. 

3. In examining the regions, the only notable data outlier is in 2012 in the West region, 

where there was a significantly higher number of SREs in comparison to the North and 

South regions, predominantly accounted for by a number of low-severity patient falls.

 Recommendations Regarding Measures in the Purchased Care Setting

a. Incorporate best practices from all three contractors to develop a more 

standardized process that enhances transparency, minimizes variation, and 

incentivizes reporting for process improvement. 

Site Visit Information

See Appendix 5.10 for core questions used to develop site visit observations. See Appendix 

Table 5.11-1 and Figure 5.16 for the total number of respondents per interview session. 

Executive Leadership Session

Executive Leadership throughout the MTFs engaged in conversation about the culture of patient 

safety within the direct care component. The Command teams provided examples of efforts to 

improve patient safety. The majority of leadership agreed that TeamSTEPPS® is recognized as 

the primary tool for reducing patient safety risk. Recognition programs such as The Good Catch 

Program have been a catalyst for increasing the volume and frequency of reporting. Other 

examples included the Patient CaringTouch System, Partnership for Patients (PfP), and 

leadership rounding, although not all commands conduct leadership rounds. Additionally, 

National Patient Safety Goals (NPSGs) and PfP guidelines to prevent injuries from falls were 

cited as safety measures in place to reduce harm. 

Functional Staff Focus Group

Patient Safety Managers (PSMs) believed that an environment of safe reporting is created by 

communicating to staff that the goal of reporting is not to assign blame, but rather to improve the 

process for the future (see Appendix 5.11). The functional staff also confirmed that public 

recognition of staff members serves as an incentive for reporting by other staff members. 

Improvements in patient safety were most effectively accomplished at facilities where a patient 

safety representative was assigned for each clinic. PSMs strive to reduce harm using myriad 

186
Military Health System Review – Final Report August 29, 2014

safety measures. Examples found include using RCA data and the Failure Modes and Effects 

Analysis tool, which is used to identify potential deficits in patient safety processes as well as to 

implement changes in systems and policies. A majority of PSMs indicated they conduct rounds 

weekly, while some stated using TJC’s tracer team concept. 

General Staff Interviews

In general, staff at sites visited indicated reporting is not a punitive matter and results are used 

for process improvement. For the most part, patient safety is accomplished by reporting the 

incident to the PSM instead of staff using the PSR tool (see Appendix Figures 5.11-1 to 5.11-8). 

When questioned regarding their role in the organization’s patient safety program, staff members 

mostly articulated three patient identifiers: falls risks evaluations, bedside rounding, and 

equipment checks for cleanliness. As a general rule, staff nurses could identify the patient safety 

roles better than any other type of staff member. Across the MTFs, TeamSTEPPS was a 

recurring theme; it was evident that it was trained and implemented extensively through the use 

of care team huddles and was a focal point for interactions with patients on a daily basis (see 

Appendix Figures 5.11-1 to 5.11-8). In describing barriers to prevent harm and PfP initiatives, 

the majority of the staff does not have a full understanding of the nine hospital-acquired 

conditions and preventable admissions as outlined in the PfP Implementation guidebook. 

Patient Interviews 

The patients throughout the MTFs visited were confident that they are receiving safe care at their 

respective facilities. Patients felt very comfortable asking questions pertaining to their care from 

not only the support staff, but also the Primary Care Managers. Not all of the patients knew the 

procedure for reporting safety issues or concerns; however, all did assert that they would report 

to someone. Patients affirmed that they consistently receive easy to follow verbal and written 

instructions with regard to their continuity care plans. 

Staff Town Hall Results

A qualitative analysis was used to evaluate the comments obtained from the staff and beneficiary 

town hall meetings. Across the MTFs, staff believes that a correlation exists between quality of 

care rendered and the culture of patient safety. Staff feels that, while it is important to provide 

high quality care and that they should strive to do so, barriers exist that prevent staff from 

providing high-quality and safe care. Appropriate staffing levels and staff mix were noted as a 

primary concern. Staff stated that increased workload due to staff shortages, as well as constant 

workforce turnover, create a sense of decreased patient care quality and safety and a lack of 

continuity of care. They also expressed that as staff rotate between departments to fill manning 

gaps, proficiency in clinical skills suffers as priority is placed on mandatory higher-directed 

training as opposed to unit-specific training.

All staff was aware of the PSR tool and its use for reporting potential; however, the majority 

expressed they did not receive feedback in a timely manner or feedback at all, rendering a 

perception of inefficiency. The cumbersome nature of using the tool made it more likely that a 

report was made verbally to a supervisor and/or safety manager rather than being submitted into 

187
August 29, 2014 5. Patient Safety in the Military Health System

the PSR tool. Furthermore, while all seven facilities indicated the importance of reporting, at 

least one member of the staff at four out of seven facilities stated that they felt they would be 

retaliated against for speaking up regarding reporting errors and events. Last, a majority of MTF 

staff shared the sentiment that the overall culture of patient safety within the direct care 

component, while adequate, has room for improvement. For example, there is a consistent 

perception from staff that leadership makes decisions in a vacuum, thereby leaving the staff 

feeling discouraged and voiceless in matters affecting delivery of care. Staff recommended that 

there be MTF-wide stand-down days to complete mandatory trainings in order to overcome its 

impact on patient care. Staff was very proud of their work and felt that they are the key drivers 

to the success of the organization.

Beneficiary Town Hall Results

Beneficiary perceptions of safe care were dominated by the availability of appointments within 

the direct care component, as well as the number of providers and support staff within the clinic. 

Patients indicated that once appointments are obtained, the care is safe. Exceptions exist in 

understaffed clinics where it is viewed that care is not thorough and staff has competing 

priorities to providing quality and safe patient care. Frequent deployment of military providers 

and subsequent changes of PCMs causes a lack of continuity of care amongst the beneficiary 

population. Moreover, while patients stated that they were comfortable asking questions of 

providers and their support staff, it was deemed futile, as the overwhelming consensus was that 

patients’ voice were not valued or heard. As far as reporting safety issues or concerns, a 

majority of patients indicated that they would report to a member of the staff; respondents at only 

one facility shared knowledge of the hospital patient advocate. Of the patients who had been 

referred to the network, a majority expressed that they received the same level of safe care as 

within the direct care component; however, respondents at one MTF indicated that the only 

reason they sought care at their respective MTF was to receive referrals to the network. As a 

whole, respondents felt that the patient safety culture in the MHS was meeting their needs based 

on their experiences in the MTF and with the network. 

Site Visit versus Central Data Comparative Summary

It is the overall assessment of the site visit team that safe and quality care is being rendered 

throughout the direct care component. While variations exist, a general consensus was found at 

all levels of the MTFs on the knowledge and practices of patient safety. Leaders encourage 

reporting of errors, near misses, and failures, and while it is apparent that staff feels comfortable 

reporting, they do so verbally to a supervisor rather than utilizing the PSR tool (see Appendix 

Figures 5.11-1 to 5.11-8). An analysis of the findings shows that while the volume of patient 

safety reporting using the PSR tool has slightly increased, this was not corroborated through 

interviews at the site visits. While the site visits indicated staff are not likely to report near 

misses if no harm comes to the patient, this was found to be inconsistent with the central data, 

which showed a slight increase in reporting. Instances were also found during staff rounds and 

town hall sessions in which employees expressed concerns regarding an environment where 

reporting was not encouraged and in fact, responses were punitive in nature. The current 

commands placed little to no emphasis on the 2011 Patient Safety Culture Survey (see Appendix 

Table 5.11-2). Some lacked knowledge of the survey, while others were not aware of the 

188
Military Health System Review – Final Report August 29, 2014

improvements made as a result of the survey by the previous command. A majority of 

commands reported, and data analysis confirmed, that the significant delay in receiving survey 

findings from the 2011 Patient Safety Culture Survey was the rate-limiting factor of a high 

priority (core interview questions) being placed on implementing change and improvements. 

Staff and patients at all MTFs addressed concerns surrounding the impact of staffing and 

workload on the level and continuity of care. This correlates with the findings of the 2011 

Patient Safety Culture Survey in which comments centered on concerns of experience and 

resources necessary for job performance.

Figure 5.16 Safety: Perceptions Among Regional Headquarters, MTF Leaders, Subject Matter 

Experts (SMEs), Staff Members, and Patients During Seven MHS Site Visits, 2014

Note: The Focus Group SMEs at the Site 1 were present during the Executive Leadership session and therefore their 

responses were counted only during the Leadership session and not the SME session.

2014 MHS Review Group 

Source: 2014 MHS Review Site Visit Survey, June - July 2014

189
August 29, 2014 5. Patient Safety in the Military Health System

Patient Safety: Overall Findings and Recommendations

1. Culture of Safety: Due to the limited number of national benchmarks in patient safety, it 

is not possible to assess whether the MHS has a culture of safety. This is evidenced by 

HSOPS, which consistently reports poor responses regarding appropriate staffing levels 

and staff mix, as well as in non-punitive response to errors and reporting. Site visits 

confirmed these findings, in that staffing and reporting of near-miss events are still areas 

of concern. Further, the Lumetra study identified reluctance in near miss reporting, and 

the review identified the lack of visibility on purchased care for patient safety. However, 

many efforts are ongoing in MTFs and DHA to identify areas for improvement and 

leadership recognizes the importance of patient safety. 

2. Policy: Neither the DoDI 6025.13 or DoDM 6025.13 define a culture of safety. The 

DoDM 6025.13 definition of a sentinel event does not provide sufficient clarity for 

consistent decision making. Moreover, it provides limited guidance on the parameters of 

a quality root cause analysis and does not include guidance on methodologies for 

capturing harm rates. Current policy requires 100-percent reporting of near miss events, 

which is unrealistic to ensure compliance.

3. Transparency: Current processes limit the ability to exchange ideas, share lessons 

learned, and increase opportunities for systemic process improvement. Site visit findings 

identified staff concerns that they did not receive feedback from events entered in the 

Patient Safety Reporting Tool. Results of root cause analysis showed that findings are 

not widely shared with frontline staff for improvement purposes. Voluntary reporting in 

the purchased care component makes comparison to the direct care system very 

challenging. There are opportunities to enhance transparency to the public through 

partnerships with patients and families.

4. Leadership: Currently there is no succinct MHS resource available for executive 

leadership to effectively advance the science and practice of quality and safety within 

their organizations. A site visit finding showed instances in which employees expressed 

concerns regarding an environment where reporting was not encouraged and in fact, the 

response to reporting was punitive in nature. HSOPS showed consistently low findings 

in organizational learning, which is a leadership responsibility. 

5. Resources: The Lumetra study recommended “the use of a single ‘closed loop’ system 

for all alerts and advisories.” Current processes limit the ability to exchange ideas, share 

lessons learned, and increase opportunities for systemic process improvement. There is 

no secure, electronic, central resource library to support daily operations for patient 

safety. The Lumetra study also recommended that the MHS “Evaluate the benefits 

versus costs of establishing permanent Patient Safety Manager (PSM) positions for 

stability.” Constraints currently exist within resource management systems, creating 

barriers to authorizing additional federal positions. There is no enterprise-wide integrated 

patient safety and quality training program. 

190
Military Health System Review – Final Report August 29, 2014

Overarching Recommendations to Improve Patient Safety

a. Implement the principles of a high reliability organization with a focus on 

leadership, culture of safety, and robust process improvement. This must be a 

strategic priority for executive leadership and will require revision of current policy 

and re-evaluation of the Patient Safety Program.

b. Re-evaluate the charter and membership of the Quality Patient Safety Risk 

Management Task Force and determine whether to use the Task Force to develop 

the framework for the HRO and submit through the existing governance structure.

c. DoD should develop a formal partnership plan with external health care 

organizations, TRICARE contractors, and national governing bodies to improve as 

a learning organization and to be at the forefront of national benchmark 

development and initiatives for patient safety.

191
August 29, 2014 5. Patient Safety in the Military Health System

This page left intentionally blank.

192
Military Health System Review – Final Report August 29, 2014

6. CONCLUSIONS

This review focused on health care access, quality of care, and patient safety in both DoDoperated and staffed health care facilities and the purchased care civilian network as operated 

through TRICARE regional contracts. A three-pronged approach was used to assess these 

aspects of care: review of enterprise-wide data and metrics; site visits of a cross-section of 

Military Treatment Facilities (MTFs) to provide local validation of centrally collected data; and 

comparison with three civilian commercial health care systems of comparable size and scope. 

Finally, nationally recognized experts conducted a review of the methodology, data, findings and 

recommendations that comprise this report (see Appendix 6.2).

The following objectives were defined in the Terms of Reference (see Appendix 1.2):

1. Assess relevant prior internal and external reports. 

2. Review policy standards and implementation.

3. Evaluate data to assess compliance and determine variance.

4. Review education and training regarding execution of policies.

5. Compare MHS performance to civilian health systems.

6. Assess the experience and perceptions of MHS patients.

7. Determine effectiveness of governance.

8. To the extent possible, identify current resources.

Access to Care

Review of policy and prior reports illustrated close alignment of policy among the Services 

without negative findings noted in prior reports. Governance has proven effective in ensuring 

consistent implementation of policy and standardization of processes across the enterprise. 

Further, education and training for access to care are well coordinated across the MHS. 

Currently available access data from the MHS revealed that a majority of patients in the direct 

care component receive care within Department of Defense (DoD) access standards. In contrast, 

data on access to care in the MHS purchased care component are not defined, collected, and 

aggregated in the same way, limiting comparability. In addition to meeting its own internal 

standards, MHS access to direct care compares favorably with that of the three external civilian 

health systems. 

Leadership at the seven facilities visited reported a strong commitment to the delivery of timely 

care. However, there were anecdotal patient reports of difficulty obtaining appointments and, at 

some facilities, staff reported limitations on same-day access due to staffing difficulties. This 

will require further review to determine specifics and significance. Several efforts are underway 

to facilitate and enhance access to care in the direct care component, to include Secure 

Messaging (with more than 1 million MTF enrollees) and the Nurse Advice Line (implemented 

across the MHS in March 2014), which handles more than 1,000 calls per day. 

193
August 29, 2014 6. Conclusions

Quality of Care

DoD policies provide substantial guidance on the quality of care program execution, but the 

MHS would benefit from specific supplemental policy. Opportunities were identified for 

improving oversight, monitoring, and communication for the quality program. These findings 

are consistent with the 2008 Lumetra study of the MHS Medical Quality Improvement Program. 

While basic education and training for quality are provided by the Services, advanced training 

and development of experts in quality of care is not routinely available.

The MHS Review Group analyzed more than 100 measures of quality of care, and identified 

performance that met or exceeded national benchmarks in many areas of inpatient and outpatient 

care; however, there are specific results that suggest underperformance and require further 

review. MHS facilities meet or exceed civilian standards for accreditation and certification, 

which validates compliance with important quality and patient safety requirements. The quality 

of care available to beneficiaries in the purchased care network is at or above the Centers for 

Medicare & Medicaid Services national averages for a wide range of conditions. Comparison of 

MHS data with that of civilian health systems demonstrated overall performance that was similar 

across a range of outpatient and inpatient measures. 

The site visit team identified a broad commitment to quality of care in all facilities visited. 

Leadership was engaged with quality initiatives and was familiar with commonly reported 

benchmarks. However, frontline staff at some facilities was not fully aware of ongoing quality 

initiatives, suggesting room for improvement. Overall, there was no clear evidence that quality 

of care was a major concern for patients; with the exception of obstetrical care, inpatient 

experience with care was highly rated.

Patient Safety

The 2008 Lumetra study made several recommendations which remain relevant, including: 

increase transparency by sharing lessons learned; establish a system to ensure feedback and 

accountability; and address variability in data reporting. Further, there is no centralized 

electronic resource to support day-to-day operations for patient safety, and no enterprise-wide 

integrated patient safety and quality training program. The new MHS governance structure 

provides the appropriate forum to address these findings. DoD has two key documents that 

provide general requirements for the patient safety program, and Service policies generally align 

with them; however, the MHS would benefit from more specific supplemental guidance. 

Assessment of the culture of patient safety in the MHS is challenging due to the limited number 

of valid metrics and national benchmarks. Results of the Hospital Survey on Patient Safety 

Culture suggest concerns with staffing levels and staff mix, as well as potentially punitive 

response to those who reported errors. Comparison with other health systems showed similar 

rates for composite safety measures, with two specific measures demonstrating potentially higher 

infection rates in the DoD. Voluntary reporting in the purchased care component makes 

comparison to the direct care component challenging. 

194
Military Health System Review – Final Report August 29, 2014

The site visit team identified staff concerns that they did not receive feedback regarding events 

entered in the Patient Safety Reporting Tool and the results of root-cause analyses. Additionally, 

there are opportunities to enhance transparency to the public through partnerships with patients 

and families. 

Summary

The MHS Review Group determined that the MHS provides safe, quality, and timely care that is 

comparable to the civilian sector. Across the enterprise, results vary by measure, both in specific 

clinical areas and at individual facilities, with a spectrum of performance ranging from high to 

low. The priority of the MHS should be identification of the causes of variance, with 

development and execution of action plans as needed. 

To be considered a leader in health care nationally, the MHS must continue its journey of 

improvement. The findings and recommendations in this report provide opportunities for further 

evaluation, analysis, and action (see Appendix 6.1 for the full list of recommendations). While 

there are more than 70 specific recommendations in the report, the following global 

recommendations lay the foundation for the MHS to focus on achieving top-tier status.

I. The MHS should identify the cause of variance for MTFs that are outliers for one or 

more measures and, when due to poor performance, develop corrective action plans to 

bring those MTFs within compliance.

II. The MHS should develop a performance management system adopting a core set of 

metrics regarding access, quality, and patient safety; further develop MHS dashboards 

with systemwide performance measures; and conduct regular, formal performance 

reviews of the entire MHS, with the DHA monitoring performance and supporting 

MHS governance bodies in those reviews.

III. The MHS should develop an enterprise-wide quality and patient safety data analytics 

infrastructure, to include health information technology systems, data management 

tools, and appropriately trained personnel. There should be clear collaboration between 

the DHA’s analytic capabilities, which monitor the MHS overall, and the Service-level 

analytic assets.

IV. The MHS should emphasize transparency of information, including both the direct and 

purchased care components, with visibility internally, externally, and to DoD 

beneficiaries. Greater alignment of measures of the purchased care component with 

those of the direct care component should be incorporated in TRICARE regional 

contracts.

V. Through MHS governance, policy guidance can be developed to provide the Services 

with common executable goals. While respecting the Services’ individual cultures, this 

effort would advance an understanding of the culture of safety and patient-centered care 

across the MHS. 

195
August 29, 2014 6. Conclusions

VI. The MHS should continue to develop common standards and processes designed to 

improve outcomes across the enterprise in the areas of access, quality, and patient 

safety where this will improve quality, or deliver the same level of quality at decreased 

cost (i.e., better value). 

196