GAO-05-659R Defense Transportation

United States Government Accountability Office

Washington, DC 20548 

September 14, 2005 

The Honorable Donald H. Rumsfeld 

Secretary of Defense 

Subject: Defense Transportation: Opportunities Exist to Enhance the 

Credibility of the Current and Future Mobility Capabilities Studies 

Dear Mr. Secretary: 

We are reviewing the processes the Department of Defense (DOD) is using to 

conduct its Mobility Capabilities Study (MCS). The MCS is to address changes in 

DOD’s transportation force structure and mobility requirements due to changes in 

threats and certain national security and military strategies. The study results 

may underpin decisions on future strategic airlift, aerial refueling aircraft, and 

sealift procurements. The study relies on the use of various models and data 

inputs to develop and evaluate transportation alternatives, including variations in 

alternative transportation modes (air, land, sea) and sources (military, civilian, 

foreign), as well as factors that affect transportation mode and source decisions. 

The Senate Armed Services Committee directed us to monitor the conduct of the 

MCS and report on the adequacy and completeness of the report no later than 30 

days after DOD completes the study.1

 As you may be aware, DOD plans to issue 

the MCS report during 2005. This letter is intended to bring to your attention 

preliminary observations on certain aspects of the MCS methodology to permit 

you to ensure the credibility of this and future studies. In our letter, we address 

the adequacy of the department’s verification, validation, and accreditation 

(VV&A) of the models and simulations being used to conduct the MCS—that is, 

the process the MCS team2

 is using to identify the models’ capabilities, limitations, 

and performance relative to the real world events they simulate. We will continue 

to monitor the MCS and will report on the adequacy and completeness of the 

methodology after DOD issues its report. 

To do our work, we reviewed applicable DOD guidance, directives, instructions, 

and memos that describe how DOD would conduct its mobility capabilities 

 

1

 S. Rep. No. 108-260, at 126 (2004). 

2

 The MCS study team includes officials from the Office of the Secretary of Defense, Program 

Analysis and Evaluation and the office of the Director of Logistics, Joint Chiefs of Staff, identified 

as co-leads and study management, as well as study participants to include representatives from 

the military services, combatant commands, and contractors employed by any of the 

aforementioned DOD organizations to provide input to or services in support of the MCS. 
Page 2 GAO-05-659R Defense Transportation 

assessments to include the National Security and Military Strategies; DOD 

Strategic Planning Guidance; DOD data collection, development, and management 

in support of strategic analysis directives; DOD modeling and simulation 

instruction; Defense Modeling and Simulation Office guidance; MCS Study Plan 

and Terms of Reference; descriptions of models used to conduct the study; and 

the databases used in the models. Additionally, we reviewed and analyzed 

previously published DOD mobility studies and past GAO reports related to the 

studies. We interviewed study officials from the Office of the Secretary of 

Defense, Program Analysis and Evaluation and the office of the Joint Chiefs of 

Staff, as well as study participants and subject matter experts from the U.S. 

Transportation Command, Air Mobility Command, Surface Deployment and 

Distribution Command, the combatant commands, and military services 

concerning the extent of their input to the study. We interviewed a modeling and 

simulation subject matter expert at the Defense Modeling and Simulation Office. 

We interviewed DOD officials to try and identify the process used to ensure the 

validation and verification of the models and the reliability of the data used in the 

study models, and conducted a comparative analysis of this process with 

applicable DOD guidance. We did not evaluate the relevancy of the DOD 

guidance because it was outside the scope of our work. We conducted this 

portion of our review from July 2004 through July 2005 in accordance with 

generally accepted government auditing standards. 

Results in Brief 

We are unable to assess the adequacy of the process DOD used to verify, validate, 

and accredit the models used to conduct the MCS. Although officials in the Office 

of Program Analysis and Evaluation stated that they have performed an equivalent 

VV&A3

 process for the models used in the MCS, there is little documentation 

available to describe the equivalent process that was used. An adequate 

evaluation of this self-described equivalent process cannot be conducted due to 

this absence of documentation, which is compounded because DOD currently 

does not plan to disclose how it conducted its equivalent VV&A process in its MCS 

report. This could negatively impact the credibility of the MCS report. DOD 

guidance requires that models and data go through a VV&A process, but officials 

in the Office of Program Analysis and Evaluation believe that this guidance is not 

relevant for models that have been used for many years, called legacy models, 

because, in their view, the models and data have already undergone an equivalent 

VV&A process consisting of actual use, although the guidance does not identify 

actual use as an appropriate equivalent process. Moreover, DOD was conducting 

VV&A on one legacy model being used in the MCS, raising questions about the 

need for such actions given the department’s statements that it is unnecessary. 

 

3

 DOD, DOD Modeling and Simulation (M&S) Verification, Validation, Accreditation (VVA), 

Instruction 5000.61 (Washington, D.C.: May 2003). 
Page 3 GAO-05-659R Defense Transportation 

We are making recommendations to improve DOD’s documentation of any 

equivalent VV&A process the department may have and to establish the relevancy 

of VV&A guidance for use with legacy models. In commenting on a draft of this 

report, DOD concurred or partially concurred with all of our recommendations. 

DOD’s comments are reprinted in their entirety in enclosure I. 

Background 

The Office of the Secretary of Defense directed its Office of Program Analysis and 

Evaluation to conduct the MCS. DOD was using an array of models and baseline 

data to develop transportation alternatives and evaluate their impact on the 

department’s capability to support military strategy. As with past mobility 

requirements studies,4

 the MCS uses a variety of models and data analyses to 

achieve the overall study objectives5

 and determine the effect of the study 

variables on the defense transportation system and its resultant effect on the 

capabilities required to meet the mobility needs for all aspects of the National 

Military Strategy. The baseline data used in the mobility models are the 

foundations for the strategic analyses and contain such data as a specific warfight 

scenario, concept of operations for the scenario, needed forces and equipment, 

battlefield terrain and weather, and time frames. According to DOD officials, the 

models have become increasingly complex over the past 15 years and are used to 

analyze large volumes of data to define mobility requirements, assess risk based 

on the forces’ ability to achieve war-fighting objectives, identify mobility gaps, and 

determine alternative methods to achieve desired capabilities. For example, the 

analysis would identify tons of equipment or number of passengers to be moved, 

the number of aircraft and ships needed to move equipment and forces, and the 

number of aircraft to be aerially refueled. 

Modeling and simulation are assuming a larger role in military assessments, 

driven in part by an appreciation for the cost, logistics, and acquisition 

implications associated with DOD programs. DOD models and simulations are to 

be developed in accordance with DOD policies, plans, and guidance.6 Generally, 

 

4

 Past mobility requirements studies conducted since the early 1990s include: Mobility 

Requirements Study (1992); Mobility Requirements Study–Bottom Up Review Update (1995); and 

Mobility Requirements Study for Fiscal Year 2005 (2001). 5

 The overall study objectives are to identify and/or quantify (1) how variations in mobility 

capabilities support the defense strategy from point of origin to point of use and return in the 2012 

time frame; (2) mobility capability gaps, overlaps, or excesses and associated risk assessments 

with regard to conducting operations; (3) mobility capability alternatives that mitigate operational 

logistic impacts caused by challenges; (4) combinations of mobility, engineering, and 

infrastructure capabilities required to support deployments and distributions required by the 

defense strategy; (5) new metrics for assessing mobility capabilities; (6) potential impact of 

evolving service force transformation and research and development efforts that integrate mobility 

concepts for the 2024 time frame; and (7) transformational accelerants to enable the defense 

transportation system to operate in a net-centric environment. 

6

 DOD, DOD modeling and Simulation Verification, Validation and Accreditation, DOD 

Instruction 5000.61 (Washington, D.C.: May 2003); and DOD Modeling and Simulation Office, DOD 
Page 4 GAO-05-659R Defense Transportation 

overall VV&A policy is established in DOD modeling and simulation master plans. 

DOD and service instructions clarify the policy and guidance indicates how to 

implement the policy. VV&A constitutes processes that gather and evaluate 

evidence to determine, based on the simulation’s intended use, the simulation’s 

capabilities, limitations, and performance relative to the real objects or events it 

simulates. 

The VV&A process entails the review, analysis, evaluation, and testing of models 

and simulations, incrementally over time as the models are being developed, by an 

independent agent or authority to improve the credibility of the process. 

Furthermore, VV&A provides enhanced user confidence, improved performance 

and reliability for the subject model and simulation results, more predictable and 

accurate modeling/simulation behavior, and reduced risk of inaccurate model 

outputs. Verification is the process of determining that a model implementation 

and its associated data accurately represent the developer’s conceptual design. 

Validation is generally understood as an independently administered process 

where multiple parties that have no vested interest in the outcomes participate in 

developing (1) an appropriate set of standard protocols for a simulation and (2) 

protocol reviews across several occasions and settings. Generally, before formal 

validation is applied, the goals of the simulation’s performance are thoroughly 

developed and specified. The validation process establishes the credibility of a 

simulation by evaluating its capability and accuracy relative to its intended use. 

Successfully completing validation enhances the credibility of the simulation by 

offering assurances that it can be relied on for reproducible results appropriate 

for its objectives. Additionally, data validation is to be put in the context of its 

suitability for use in models. For this reason, the data cannot be validated 

independently of the models for which they are intended. The results of the 

verification and validation phase are used to support the accreditation decision, 

which is the user’s official certification that a model, simulation, or federation of 

models and simulations and the associated data are acceptable for use for a 

specific purpose. 

Documentation of VV&A Process Is Lacking 

Officials in the Office of Program Analysis and Evaluation stated that they have 

performed an equivalent VV&A process for the legacy models used in the MCS, 

but there is little documentation available to describe the equivalent process that 

was used. As a result, we are not able to assess the adequacy of DOD’s selfdescribed equivalent process. Office of Program Analysis and Evaluation and 

Joint Data Support officials told us that most of the documentation does not yet 

exist and will not be completed until after the MCS is completed. Moreover, at the 

time of our review, DOD had not planned to describe the equivalent VV&A 

process in its MCS report. The absence of VV&A documentation for the models 

 Verification, Validation and Accreditation Recommended Practices Guide (Washington, D.C.: 

August 2004). 
Page 5 GAO-05-659R Defense Transportation 

and data used to conduct the MCS and the lack of disclosure in the published MCS 

report could limit the study’s credibility. 

DOD guidance, issued by the Under Secretary of Defense for Acquisition, 

Technology and Logistics, requires that DOD models and data go through a VV&A 

process. The Office of Program Analysis and Evaluation acknowledged that it did 

not comply with the guidance because it believes such an approach is not 

warranted for legacy models that have been used for many years. Moreover, these 

officials believe that such long-term use constitutes a VV&A process equivalent to 

that required in the DOD guidance. However, the DOD guidance does not identify 

the actual use of a model as constituting an equivalent VV&A process. 

DOD is using the following nine mobility models to conduct the MCS: 

• Aerial Port of Debarkation 

• Air Mobility Operations Simulation 

• Combined Mating and Ranging Planning System 

• CONUS (Continental U.S.) Enhanced Logistics Intra-theater Support Tool 

• Model for Inter-theater Deployment by Air and Sea 

• Integrated Computerized Development System 

• Joint Throughput Modeling Tool 

• TRANSPORT 

o Airlift/Sealift Throughput Tool 

o Airlift/Sealift Rapid Analysis Tool 

o Airlift Simulation Tool & Seaport Simulation Tool, and 

• Analysis of Mobility Platform Federation. 

According to DOD officials, eight of the nine models did not go through the 

VV&A process specified in the DOD guidance.7

 Office of Program Analysis and 

Evaluation, U.S. Transportation Command, Air Mobility Command, and Surface 

Deployment and Distribution Command officials told us that (1) most of the 

models have been used within DOD for many years and have proved reliable, and 

(2) many subject matter experts work with the models and the output daily to 

assure ongoing error detection and swift corrections when needed. These 

officials maintain that actual use of the models for a long period of time 

constitutes an equivalent VV&A process. 

Nonetheless, DOD officials were conducting VV&A on one of the legacy models 

being used in the MCS while MCS officials were simultaneously questioning the 

relevancy of DOD’s guidance for the legacy models. For example, Surface 

Deployment and Distribution Command officials told us that the CONUS 

Enhanced Logistics Intra-theater Support Tool model was undergoing VV&A at 

the time of our review. It is unclear why DOD is conducting VV&A on this model 

 

7

 DOD conducted VV&A on the Combined Mating and Ranging Planning System model about 15 

years ago in compliance with departmental guidance but could not locate documentation to 

demonstrate how VV&A was done. As a result, we could not evaluate the adequacy of the VV&A 

process. 
Page 6 GAO-05-659R Defense Transportation 

given Office of Program Analysis and Evaluation and other officials’ belief that it 

is unnecessary. Furthermore, the extent to which DOD’s guidance may be 

irrelevant as asserted by these officials is unknown, because the department has 

not evaluated the current VV&A guidance to determine its relevancy for use with 

legacy models. 

Conclusions 

Models and simulations approximate the real world. The approximations must be 

justified to assure modeling and simulation users that their predictions are 

credible within the bounds of specific situations, environments, and 

circumstances. When modeling and simulation are credible, decision makers 

have greater assurance that they are well informed and thus can make wellfounded decisions. VV&A reduces the risk inherent in the use of models and 

simulations by improving the credibility of modeling and simulation results. 

VV&A also enhances credibility by applying a process of incremental review, 

analysis, evaluation, and testing by an independent agent. In light of the fact that 

DOD did not follow its guidance, the absence of model and baseline data VV&A 

documentation for the models and data used to conduct the MCS, and the planned 

lack of disclosure in the soon to be published MCS report, could limit the study’s 

credibility. Moreover, MCS officials maintain that DOD guidance regarding VV&A 

is not relevant to legacy models and data. However, until the department 

evaluates the guidance to determine its relevancy for use with legacy models or 

incorporates guidance showing how actual model usage is to be applied as an 

equivalent VV&A process, the validity of DOD’s assertion is uncertain. When 

conducted as intended, VV&A provides greater assurance that the MCS outputs 

are accurate. Ultimately, if the MCS inaccurately identifies mobility requirements, 

DOD officials may be less well informed and may therefore inadvertently obtain 

insufficient mobility assets or more than needed and thus waste resources. 

Recommendations for Executive Action 

We recommend that you direct the Director, Office of Program Analysis and 

Evaluation, to take the following three actions: 

• develop documentation that describes the equivalent VV&A process used 

to verify and validate the mobility models and baseline data used to 

conduct the MCS prior to publishing any portion of the study results, 

• disclose in the published MCS report the equivalent VV&A process used on 

the models and baseline data, and 

• work with the Office of the Under Secretary of Defense for Acquisition, 

Technology and Logistics to evaluate the current DOD VV&A guidance to 

determine its relevance for use with legacy models and to change the 

guidance if appropriate. 
Page 7 GAO-05-659R Defense Transportation 

Agency Comments and Our Evaluation 

In commenting on a draft of this report, the DOD concurred with two of our 

recommendations and partially concurred with the third. 

In its comments, DOD concurred with our first and second recommendations that 

the Office of Program Analysis and Evaluation develop documentation that 

describes the equivalent VV&A process used to verify and validate the mobility 

models and baseline data used to conduct the MCS prior to publishing any portion 

of the study results, and disclose in the published MCS report the equivalent 

VV&A process used on the models and baseline data. In its comments, DOD 

stated that the mobility modeling community has amassed substantial expertise 

during the past 25 years and that a “vigorous, collaborative VV&A process that is 

fully consistent with and in many respects exceeds the intent of DOD VV&A 

guidance has been put into place.” DOD noted the MCS report will provide 

information on the VV&A process, and will exceed the level of documentation 

provided in past reports on DOD mobility studies. 

DOD partially concurred with our recommendation to evaluate the current DOD 

VV&A guidance to determine its relevance for use with legacy models and change 

the guidance if deemed appropriate. In its response, DOD essentially agreed with 

our recommendation, but pointed out that the Office of the Under Secretary of 

Defense for Acquisition, Technology and Logistics is responsible for VV&A 

guidance. Therefore, we refined our recommendation to recommend that the 

Director of the Office of Program Analysis and Evaluation work with the Office of 

the Under Secretary of Defense for Acquisition, Technology and Logistics to 

evaluate the current DOD VV&A guidance to determine its relevance for use with 

legacy models and to change the guidance if appropriate. 

Additionally, DOD expressed concern that it was premature to reach a conclusion 

as to the adequacy of the department’s VV&A process because some of the 

documentation requested cannot be provided until the final report is written and 

that our draft report should be amended. We agree that we cannot assess the 

adequacy until the documentation is available and have so stated in our report. 

We also stated in our report that at the time of our review, DOD had not planned 

to describe the VV&A process in its MCS report nor had it planned to perform 

VV&A because the legacy models being used were reliable. As we noted in our 

report, at least one of the models was already undergoing a VV&A. 

Moreover, DOD questioned our conclusion that if an adequate VV&A process 

could not be documented, then the credibility of the MCS could be limited. 

Specifically, DOD stated that convincing empirical evidence indicates that the 

MCS report’s credibility will not be limited by the VV&A documentation 

associated with the legacy models, because DOD has used the models for two 

decades with no credibility limitations noted. As we noted in our report, DOD 

guidance indicates that a well-documented VV&A process for the models used to 
Page 8 GAO-05-659R Defense Transportation 

conduct the study will add to the MCS report’s credibility. Also as we noted in our 

report, DOD guidance states that the VV&A process provides enhanced user 

confidence, improved performance and reliability for the subject model and 

simulation results, reduced risk of inaccurate model outputs, and offers 

assurances that a particular model or simulation can be relied on for reproducible 

results appropriate for its objectives. While we support the use of empirical 

evidence, such evidence is normally verifiable. 

Finally, DOD expressed concern that we linked the VV&A of mobility models used 

to conduct the MCS with the prospect of inaccurate identification of mobility 

requirements. We disagree. DOD guidance states that VV&A reduces the risk 

inherent in the use of models and simulations by improving the credibility of their 

results and provides greater assurance that the study outputs are accurate. If the 

models supporting the MCS do not effectively simulate the real world and DOD 

uses the results to complete the MCS and subsequently base acquisition decisions, 

then we maintain our caution--DOD could inadvertently obtain insufficient 

mobility assets or could acquire more than needed and thus waste resources. 

Enclosure I contains the full text of DOD’s comments. 

— — — — 

We are sending copies of this report to the Chairman and Ranking Members of the 

Senate Armed Services Committee as well as to the Chairmen and Ranking 

Members of the Senate Appropriations Committee, Subcommittee on Defense, the 

House Armed Services Committee, and the House Appropriations Committee, 

Subcommittee on Defense. This letter is also available at no charge on the GAO’s 

Web site at http:www.gao.gov. If you or your staff have any questions on the 

matters discussed in this letter, please contact me at (202) 512-8365 or 

solisw@gao.gov. Key contributors to this report are listed in enclosure II. 

Sincerely yours, 

William M. Solis, Director 

Defense Capabilities and Management 

Enclosures
Enclosure I 

Page 9 GAO-05-659R Defense Transportation 

Comments from the Department of Defense 
Enclosure I 

Page 10 GAO-05-659R Defense Transportation 
Enclosure I 

Page 11 GAO-05-659R Defense Transportation 
Enclosure II 

Page 12 GAO-05-659R Defense Transportation 

GAO Contact and Staff Acknowledgments 

GAO Contact William M. Solis, (202) 512-8365 

Acknowledgments Key contributors to this report include Brian J. 

Lepore, Claudia Dickey, Ron La Due Lake, 

Oscar Mardis, Deborah Owolabi, Kenneth 

Patton, and R.K. Wild. 

(350708)