Defense Transportation: Study Limitations Raise Questions about the Adequacy and Completeness of the Mobility Capabilities Study and Report
Published: 2006-09-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) issued the Mobility Capabilities Study (MCS), which was intended to identify and quantify the mobility capabilities needed to support U.S. strategic objectives into the next decade. The MCS found that projected capabilities are adequate to achieve U.S. objectives with an acceptable level of risk--that is, current U.S. inventory of aircraft, ships, prepositioned assets, and other capabilities are sufficient, in conjunction with host nation support, and assuming planned investments take place. The Senate report accompanying the bill for the fiscal year 2005 Defense Authorization Act required GAO to report on the adequacy and completeness of the MCS. GAO assessed the extent to which the MCS met generally accepted research standards that this type of study would be expected to meet to be considered sound and complete.
section_title: What GAO Found
paragraphs: DOD used an innovative approach in conducting the study and acknowledged methodological limitations in its report; however, it did not fully disclose how these limitations could affect the MCS conclusions and recommendations. Therefore, it is not transparent how the analyses done for the study support DOD's conclusions. Measured against relevant generally accepted research standards, GAO has identified limitations in the MCS and its report that raise questions about their adequacy and completeness. GAO suggests that Congress and other decision makers exercise caution in using the MCS to make investment decisions. Among GAO's findings: Aspects of modeling and data were inadequate in some areas because data were lacking and the models used could not simulate all relevant aspects of the missions. The report did not explain how these limitations could affect the study results or what the impact on projected mobility capabilities might be. Generally accepted research standards require that models used are adequate for the intended purpose, represent a complete range of conditions, and that data used are properly generated and complete. For example, the MCS modeled hypothetical homeland defense missions rather than homeland defense demands derived from a well defined and approved concept of operations for homeland defense, because the specific details of the missions were still being determined and the data used may be incomplete. The MCS also was unable to model the flexible deterrent options/deployment order process to move units and equipment into theater because of lack of data, but the study assumed a robust use of this process. In addition, the MCS report contains over 80 references to the need for improved modeling or data. While the MCS concluded that combined U.S. and host nation transportation assets were adequate, in describing the use of warfighting metrics in its analyses, the report does not provide a clear understanding of the direct relationship of warfighting objectives to transportation capabilities. Additionally, the report stated that further analysis is required to understand the operational impact of increased or decreased strategic lift on achieving warfighting objectives. Relevant generally accepted research standards require that conclusions be supported by analyses. The use of both warfighting and mobility metrics would allow decision makers to know whether combat tasks were achieved and how much strategic transportation is needed to accomplish those tasks. In some cases, the MCS results were incomplete, unclear, or contingent on further study, making it difficult to identify findings and evaluate evidence. Relevant research standards require results to be presented in a complete, accurate, and relevant manner. For example, the report contains recommendations for further studies and assessments, five of which are under way. However, DOD has no plans to report the impact of these studies on the MCS results after the studies are complete. In addition, the report contains qualified information that is not presented clearly, such as varying assessments of intra-theater assets in three different places.

section_title: Letter
paragraphs: The National Security Strategy of the United States requires global  mobility through rapid, effective, and efficient projections of power at  home and abroad to deploy and sustain America’s armed forces. To  improve its mobility capabilities, the Department of Defense (DOD) plans  to spend more than $50 billion from fiscal years 2006 through 2011 for  aircraft, ships, ground transportation, prepositioned assets, and other  mobility assets. DOD has conducted several studies to determine mobility  requirements and recently completed a study of its mobility capabilities  and issued a report in December 2005. The intent of the Mobility  Capabilities Study (MCS) was to identify and quantify the mobility  capabilities needed to support U.S. strategic objectives into the next  decade. The MCS determined that the projected mobility capabilities are  adequate to achieve U.S. objectives with an acceptable level of risk during  the period from fiscal years 2007 through 2013; that is, the current U.S.  inventory of aircraft, ships, prepositioned assets, and other capabilities are  sufficient, in conjunction with host nation support. The MCS emphasized  that continued investment in the mobility system, in line with current  departmental priorities and planned spending, is required to maintain  these capabilities in the future. This includes, for example, fully funding  Army prepositioned assets as planned and completing a planned  reengineering of the C-5 aircraft. The MCS report also made  recommendations to conduct further studies, develop plans and strategies,  and improve data collection and mobility models. In fact, DOD officials  told us that a Mobility Capabilities Study--2006 is underway.
In the Senate report accompanying the bill for the fiscal year 2005 Defense  Authorization Act, you asked us to monitor the process used to conduct  the MCS and report on the adequacy and completeness of the study.  Specifically, our objective was to determine whether the MCS was  adequate and complete. On March 1, 2006, we briefed your staff on our  preliminary observations. This report expands on that briefing and makes  recommendations to the Secretary of Defense.
To conduct our review of the MCS, we analyzed the final MCS report, the  MCS Terms of Reference and MCS Study Plan, as well as other DOD  policies and guidance concerning how DOD would conduct the MCS and  the databases and models used in the study. We identified generally  accepted research standards that define a sound and complete quality  study that were relevant to the MCS, and assessed the extent to which the  MCS report met these standards. We interviewed study officials, study  participants, and subject matter experts from several DOD entities,  including the combatant commands and the military services. As we  monitored the development of the MCS, we requested that DOD provide  documentation supporting and verifying key analytical and decision- making processes. DOD officials could not produce this documentation  during the development of the MCS or following issuance of the report.  Consequently, we were unable to fully determine whether the analytical  and decision-making processes that we believe are significant to the  credibility of the study supported the MCS effort and its conclusions. Our  scope and methodology are discussed in more detail in appendix I. We  conducted our work from July 2004 through July 2006 in accordance with  generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The MCS was the first assessment of DOD’s mobility system since 2000.  The study was designed to identify changes in DOD’s transportation force  structure due to changes in threats and national security and military  strategies. The MCS is the fourth in a series of major mobility studies that  DOD has conducted since the end of the Cold War. The first study, the  Mobility Requirements Study, conducted in 1992, was undertaken because  of concern about the DOD’s strategic mobility capabilities in the wake of  Operation Desert Shield and Operation Desert Storm. That study  established mobility requirements for the post-Cold War era; defined  baseline requirements for intertheater, or strategic, mobility; and proposed  a long-range investment plan to meet these requirements. The Mobility  Requirements Study Bottom-Up Review Update, conducted in 1994,  reaffirmed the need for increases in key mobility components and  validated the prior study’s recommendation for the procurement of  additional ships for afloat prepositioning and for surge deployments of  forces based in the continental United States. The Mobility Requirements  Study—2005, issued in 2001, projected future mobility requirements based  on two nearly simultaneous major regional contingencies. It included a  broader range of factors, including host nation support and enemy use of  weapons of mass destruction, than the previous studies.
The current MCS, which began in May 2004, reassessed DOD’s mobility  capabilities against the backdrop of a revised National Military Strategy  that included the ongoing war against violent extremism, an evolving  global defense posture, a new force-sizing construct, revised campaign  scenarios, and ongoing departmentwide transformation efforts. The study  results were intended to support decisions on future strategic airlift, aerial  refueling aircraft, and sealift procurements needed to meet varying  military requirements. The study used an innovative “capabilities-based”  approach, measuring existing and currently projected mobility capabilities  against warfighting demands that could be expected in fiscal year 2012  while also considering mobility demands during the 7-year period from  fiscal year 2007 through fiscal year 2013. According to DOD officials, the  Secretary of Defense believed this approach would give him greater  flexibility in deciding which capabilities to fund in a constrained budget  environment. In considering each aspect of the National Military Strategy,  the MCS modeled warfighting scenarios in the year 2012 using different  premises with varying assumptions to develop and evaluate mobility  capability mix alternatives. The models were used to evaluate  transportation alternatives, including variations in alternative  transportation modes (air, land, sea) and sources (military, civilian,  foreign), as well as factors that affect transportation mode and source  decisions. The scope of the MCS described the study as an assessment of  the full range of transportation needs required to support (1) combat  operations; (2) smaller military efforts, such as peacekeeping or overseas  evacuation of American citizens; (3) homeland defense/civil support, such  as disaster relief and antiterrorism response; and (4) other strategic  missions, such as strategic nuclear and global strike missions. The study  was coauthored by the Office of the Secretary of Defense, Office of the  Director, Program Analysis and Evaluation (PA&E), and the Chairman,  Joint Chiefs of Staff, Office of the Director of Logistics. Other DOD  components involved in the study included the U.S. Transportation  Command and its subordinate commands, the Office of the Under  Secretary of Defense (Acquisition, Technology and Logistics), the  combatant commanders, the military services, and others. The final report  was signed on December 19, 2005, by the Deputy Secretary of Defense.
section_title: Limitations in the MCS Study and Report Raise Questions about Adequacy and Completeness
paragraphs: As measured against relevant generally accepted research standards,  limitations in the MCS study and report raise questions about their  adequacy and completeness. For example, aspects of modeling and data  were inadequate in some areas because data were lacking and some of the  models used could not simulate all relevant aspects of the missions.  Furthermore, the exclusive use of warfighting metrics in the MCS analyses  limited the usefulness of the report. Moreover, in some cases the MCS  results were incomplete, unclear, or contingent on further study, making it  difficult to identify findings and evaluate evidence. Finally, verification,  validation, and accreditation of models and data used to conduct the study  were incomplete because they were not done in accordance with DOD  policy or relevant research standards, and supporting documentation for  key processes could not be provided.
subsections: section_title: Aspects of Modeling and Data Were Inadequate
paragraphs: Aspects of modeling and data were inadequate in some areas because data  were lacking and some of the models used could not simulate all relevant  aspects of the missions. Relevant research standards require that models  used are adequate for the intended purpose, represent a complete range of  conditions, and that data used are properly generated and complete. As  DOD acknowledged in the MCS report as a study limitation, some  modeling tools were not available to analyze key missions. The MCS cited  deficiencies in several existing mobility models and the need for follow-on  MCS analyses. The MCS report also acknowledged that the identified  deficiencies in data hindered analysis of future requirements and  transportation system performance. However, the report did not explain  how these limitations could affect the study results or what the effect on  the projected mobility capabilities might be.
For example, the MCS modeled hypothetical homeland defense missions  rather than homeland defense demands derived from a well defined and  approved concept of operations for homeland defense, because the  specific details of the missions were still being determined, and DOD  acknowledged that the data used are incomplete. The MCS report  recommended further analysis of mobility capabilities after homeland  defense needs are refined. However, the report did not identify the  potential effect that using these hypothetical scenarios might have on the  MCS results. The MCS also was unable to model the flexible deterrent  options/deployment order process to move units and equipment into  theater because of lack of data on how deployment orders have been  issued in the past for major combat operations. However, the MCS  assumed a robust use of the flexible deterrent option/deployment order  process, which in one scenario accounted for approximately 60 percent of  the early airlift movement prior to the beginning of combat operations.  Instead, the MCS modeled the flow of forces and equipment contained in  the time-phased force deployment data process. Based on the scenarios  provided for the MCS analyses, we could not determine how the  deployment order process would affect the mobility assets required for  major combat operations. The MCS report noted that additional analysis is  required to determine the implications of the deployment order process  and to provide sufficient information for decision makers concerning the  amount of future mobility assets actually needed.
In addition to these modeling and data issues, the MCS report contains  more than 80 references to the need for improved modeling, and 12 of  these references call for additional data or other refinements, such as  follow-on analyses, further assessments, future evaluations, additional  study, and investigation of alternatives in a wide range of areas, such as  antiterrorism response, infrastructure availability, intratheater airlift,  strategic sealift, air refueling, and availability of civil reserve aircraft.  Some of these studies are currently underway, as discussed later in this  report.
Moreover, our analysis of the MCS report showed that the year modeled  (2012) to determine the DOD transportation capabilities needed for the  years 2007 through 2013 did not place as much demand for mobility assets  in support of smaller military operations, such as peacekeeping, as other  years. To establish transportation demands for mission requirements, DOD  developed and used a baseline security posture that covered a 7-year  period. This baseline was developed, in part, using a historical analysis of  DOD’s movement of personnel, equipment, supplies, and other items.  According to DOD officials, Office of the Under Secretary of Defense for  Policy, which developed the baseline security posture, selected the year  modeled in the MCS because it was deemed the “most likely” to occur in  terms of transportation demands and because it was not statistically  different from other years in the 7-year period. However, our analysis  showed that 2012 involved the least demand for transportation assets in  support of smaller military efforts than any year in the 7-year period and  did not fully stress the defense transportation system. Figure 1 depicts the  number of hypothetical ongoing contingencies for each year in the  baseline as shown in the MCS.
Although not transparent in the study, DOD officials said scenarios in the  year modeled were not intended to fully stress the defense transportation  system. DOD officials provided no further explanation for the year  selected to develop the DOD transportation capabilities other than it was  directed by Office of the Under Secretary of Defense for Policy and agreed  to by the study leadership. We believe that selection of a different year that  placed an increased demand on transportation assets for smaller military  efforts may have revealed gaps in mobility requirements. Therefore, we  found that the selection of 2012 as the modeling year was a limitation in  the MCS with respect to smaller military efforts.
Because of these modeling and data limitations, the MCS may have  incorrectly estimated the future mobility requirements needed to support  homeland defense missions, major combat operations, and smaller  contingencies. Until DOD improves aspects of the modeling and data used  to conduct the MCS—to include defining its homeland defense mission  needs, developing models for the deployment order process, and  explaining how identified modeling and data limitations could affect the  study results—decision makers may not have adequate and complete  information about DOD’s mobility capabilities.
section_title: Exclusive Use of Warfighting Metrics in MCS Analyses Limited Usefulness of Report
paragraphs: While the MCS concluded that combined U.S. and host nation  transportation assets were adequate to meet U.S. objectives with  acceptable risk, the report, in describing the use of warfighting metrics in  its analyses, does not provide a clear understanding of the direct  relationship of warfighting objectives to transportation capabilities. The  report acknowledged that further analysis is required to understand the  operational impact of increased or decreased strategic lift on achieving  warfighting objectives. Relevant generally accepted research standards  require that conclusions be supported by analyses. The use of warfighting  metrics is a measure to determine whether combat tasks, such as  establishing air superiority, are achieved. However, they do not measure  success in terms of whether appropriate personnel, supplies, and  equipment arrived in accordance with timelines. As a result, we could not  determine how the study concluded that planned transportation assets  were adequate because the study did not contain a transparent analysis to  support its conclusion. In our opinion, it is important for decision makers  to have an understanding of both combat tasks that must be achieved and  the amount of transportation assets needed to achieve those tasks with  some level of success. This understanding would allow creation of a clear  roadmap for investment decisions. However, we could not determine how  the study calculated the specific numbers of transportation assets needed  or whether there are specific gaps, overlaps, or excesses in transportation  assets, a key objective of the study. Previous DOD mobility studies,  including the Mobility Requirements Study—2005, primarily used mobility  metrics, which measured success in terms of tons of equipment and  personnel moved per day to accomplish military objectives. Million-ton- miles per day is a commonly accepted measure of airlift performance and  reflects how much cargo can be delivered over a given distance in a given  period of time based on the capability of each type of aircraft. A similar  mobility metric—short tons—is used to measure ship capability. However,  these studies did not fully integrate combat tasks as a metric. The use of  both warfighting and mobility metrics to measure success would allow  decision makers to know whether there is sufficient capability to achieve  warfighting objectives, as well as to understand the number, type, and mix  of mobility assets that are actually needed.
section_title: Results Are Not Always Complete or Presented Clearly and Are Qualified or Contingent on Further Study or Analysis
paragraphs: In some cases, the MCS results were incomplete, unclear, or contingent on  further study, making it difficult to identify findings and evaluate evidence.  Relevant research standards require results to be presented in a complete,  accurate, and relevant manner; conclusions to be sound and complete; and  recommendations to be supported by analyses. Our analysis of the MCS  report found that it contains several recommendations for further studies  and assessments, five of which are under way. The five studies address  intratheater lift capabilities; sealift petroleum, oil, and lubricants; logistics  contingency operations capabilities; aerial refueling; and integrated global  presence and basing. However, the report does not explain the potential  effect of these ongoing studies on the MCS results after the studies are  complete, nor does DOD have plans to report the effect of these studies on  the MCS results.
In addition, the report contains qualified information that is not presented  clearly in the report, such as varying assessments of intratheater assets in  three different places. For example, the report states in the assessment  section of the executive summary that projected transportation assets are  sufficient to address intratheater demands in the fiscal years 2007 through  2013 time frame. However, in the recommendations section of the  executive summary, the report states that DOD should take action to  determine the proper mix of intratheater assets needed to meet  requirements. Then, in the part of the report that discusses intratheater  airlift, the report states that a detailed analysis of intratheater airlift needs  would require improved modeling tools to accurately capture interactions  among land, sealift, and airlift capabilities and that data sets must be  developed that accurately describe the requirement in light of emerging  concepts.
section_title: VV&A of Models and Data Was Not Complete
paragraphs: VV&A of models and data used to conduct the study was not complete  because it was not done in accordance with DOD policy or relevant  research standards. DOD policy issued by the Under Secretary of Defense  for Acquisition, Technology and Logistics requires that DOD models and  data go through a VV&A process. Moreover, relevant research standards  state that a study report should include a VV&A accreditation report that is  signed by the study director and addresses the models and data  certification. DOD officials acknowledged that they did not comply with  the VV&A policy when using legacy models in the MCS because they  believed such an approach was not warranted for legacy models that have  been used for many years and have proved reliable. Moreover, these  officials believe that such long-term use constitutes a VV&A process  equivalent to that required in the policy. However, the DOD policy does  not specify that the actual use of a model constitutes an equivalent VV&A  process. VV&A of models and data reduces the risk inherent in the use of  models and simulations by improving the credibility of modeling and  simulation results. We previously reported our concerns that DOD did not  follow its policy in executing the MCS and had little documentation to  support the VV&A process used. We found that the final MCS report  contained a description of the equivalent VV&A process, but DOD officials  could provide no further documentation to verify and validate this  equivalent process other than the description included in the report. We  also found no documentation in the study report to support DOD claims  that the models have proven reliable.
Furthermore, DOD officials were unable to provide documentation to  support and verify key analytical and decision-making processes used by  senior DOD leadership throughout the study. Relevant research standards  support documenting the study’s analytical and decision-making processes  to ensure the accuracy, completeness, and credibility of study results.  DOD officials told us that the study’s key analytical and decision-making  processes were validated and approved by study participants during  working group meetings and by senior leadership during General Officer  Steering Committee meetings and Executive Committee meetings. PA&E  officials could not produce documentation of these meetings because they  said documentation did not exist. Nor could they produce other  documents we requested during the development of the MCS or following  issuance of the report. Consequently, we were unable to determine the  adequacy and completeness of the analytical and decision-making  processes that supported the MCS effort and that we believe are  significant to the credibility of the study and its conclusions.
section_title: Conclusions
paragraphs: The methodological limitations in the MCS that we identified—some of  which were acknowledged by DOD in the MCS report—raise questions  about the adequacy and completeness of the study and its report. Until  DOD improves aspects of the modeling and data used to conduct the  MCS—such as defining its homeland defense mission needs and  developing models for the deployment order process—decision makers  may not have adequate and complete information about DOD’s mobility  capabilities to enable them to make fully informed investment decisions.  Furthermore, in the absence of an explanation of how identified modeling  and data limitations could affect the study results or how such limitations  could affect projected mobility capability requirements, the accuracy of  the study’s finding that projected capabilities are adequate to achieve U.S.  objectives with an acceptable level of risk during the period from fiscal  years 2007 through 2013 is unclear. Moreover, without a transparent  comparison between existing mobility assets and projected needed assets,  decision makers will be unable to use study results to identify and quantify  the specific types and mix of mobility assets needed to address mobility  gaps, overlaps, and excesses. Until DOD conducts an adequate and  complete future MCS and clearly discloses all limitations and their effects  on the study results, decision makers likely will not have full information  concerning DOD’s mobility capabilities. As a result, we suggest that  Congress and other decision makers exercise caution in using the MCS to  make programmatic investment decisions.
section_title: Recommendation for Executive Action
paragraphs: To provide decision makers with adequate and complete information  concerning mobility capabilities so they are able to clearly understand the  operational implications of the study and make fully informed  programmatic investment decisions, and to improve the usefulness of  future mobility capabilities studies, we recommend that the Secretary of  Defense take the following three actions, when conducting future mobility  capabilities studies beginning with any study currently underway:    develop models and data for all critical missions, such as homeland  defense, and processes, such as the flexible deterrent options/deployment  order process;  include in study reports an explanation of how stated limitations might  impact the study results and, at a minimum, describe how recommended  future studies might be conducted to enhance the results of the original  study; and  incorporate both mobility and warfighting metrics in determining  capabilities.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, DOD concurred with our first and  third recommendations. DOD stated it did not understand our second  recommendation that the Secretary of Defense, when conducting future  mobility studies, beginning with any study currently underway, include in  study reports an explanation of how ongoing and follow-on studies and  modeling and data limitations that are referenced in the report could affect  the reported results. DOD also noted that it plans to continue its ongoing  efforts to enhance the models and data collection processes used to assess  mobility capabilities across the full range of strategic missions and  supports the notion that continual improvements are needed. As we noted  throughout our report, the MCS report contains numerous and repeated  references to the need for improved modeling and additional data or other  refinements, such as follow-on analyses, further assessments, future  evaluations, additional study, and investigation of alternatives in a wide  range of areas. DOD further commented that while a completed study can  recommend that follow-on studies be conducted, it cannot explain how  future studies might affect the results of the current study. We  acknowledge that DOD cannot quantitatively predict the outcome of an  ongoing or future study. However, we believe DOD should be able to  explain what ongoing follow-on studies or evaluations seek to determine,  what changes are being made to the data inputs and modeling tools that  are being used to conduct the studies, and how DOD expects the results  may differ from current study results. While the explanation may be  hypothetical, as are many operations research study hypotheses, it can  provide decision makers with a better understanding of the current study’s  limitations and results and how an ongoing or future study’s results may  differ. Therefore, we refined our recommendation to recommend that the  Secretary of Defense, when conducting future mobility studies, beginning  with any study currently under way, include in study reports an  explanation of how stated limitations might impact the study results and,  at a minimum, describe how recommended future studies might be  conducted to enhance the results of the original study. For example, if  modeling and data are limitations to a study, the report should discuss the  ways in which the results might change with better models and data.
DOD also commented that our report contained misleading information  and factual errors and that it stands by the adequacy and completeness of  the MCS.  The department provided examples in its technical comments  where it believed our report contained misleading information and factual  errors.  We disagree with the department’s comments regarding the facts  in our report and have addressed each of the department’s comments in  appendix II.
Lastly, DOD stated that the MCS and its conclusions are well accepted by  the civilian and military leadership of the department, and pointed out that  in March 2006 testimony before the House Armed Services Committee, the  Commander, U.S. Transportation Command, stated that the planned  strategic airlift fleet determined by the MCS is “about the right capacity”.  However we note that in the same hearing, the Commander also stated  that he thought DOD needed “somewhere in the neighborhood of” 20 C-17  cargo aircraft beyond what is planned. We also note that in the  Commander’s April 2006 testimony before the Senate Armed Services  Committee, Subcommittee on Sealift, he stated that, in an internal  Focused Mobility Analysis to study strategic mobility from a  Transportation Command perspective, the MCS will be the baseline, “but  we will explore how changes in key assumptions may impact the  analytical outcome.”
We are sending copies of this report to the Secretary of Defense; the  Director of PA&E; and the Office of the Chairman, Joint Chiefs of Staff. We  will also make copies available to others upon request. In addition, the  report is available at no charge on the GAO Web site at  http://www.gao.gov. If you or your staff have any questions regarding the  briefing or this report, please contact me at (202) 512-8365 or  solisw@gao.gov. Contact points for our Offices of Congressional Relations   and Public Affairs may be found on the last page of this report. GAO staff  who made key contributions to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To conduct our review of the Mobility Capabilities Study (MCS), we  reviewed and analyzed the final MCS report; the MCS Terms of Reference;  the MCS Study Plan; applicable Department of Defense (DOD) strategic  planning guidance; as well as other DOD guidance, directives, instructions,  and memos that describe how DOD would conduct its MCS. We also  reviewed the National Security Strategy of the United States of America  and the National Military Strategy of the United States of America; DOD  guidance concerning data collection, development, and management in  support of strategic analysis; DOD modeling and simulation instruction;  Defense Modeling and Simulation Office guidance; descriptions of models  used to conduct the study; and the databases used in the models. We  interviewed study officials from the Office of the Secretary of Defense,  Program Analysis and Evaluation (PA&E), and the office of the Chairman,  Joint Chiefs of Staff, Logistics, as well as study participants and subject  matter experts from the U.S. Transportation Command, Air Mobility  Command, Surface Deployment and Distribution Command, the  combatant commands, and the military services concerning the extent of  their input to the study. We also interviewed officials from the Office of  the Secretary of Defense, Acquisition, Technology and Logistics, and the  Modeling and Simulation Technical Director at the Defense Modeling and  Simulation Office.
Additionally, we reviewed research literature and DOD guidance and  identified frequently occurring, generally accepted research standards that  are relevant for defense studies such as the MCS that define a quality or  sound and complete study. The following were our sources for these  standards:    GAO, Government Auditing Standards: 2003 Revision, GAO-03-673G  (Washington, D.C.: June 2003);    GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.:    GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.: February    RAND Corporation, RAND Standards for High-Quality Research and  Analysis (Santa Monica, Calif.: June 2004);    Air Force Office of Aerospace Studies, Analysts Handbook: On  Understanding the Nature of Analysis (January 2000);    Air Force, Office of Aerospace Studies, Air Force Analysis Handbook, A  Guide for Performing Analysis Studies: For Analysis of Alternatives or  Functional Solution Analysis (July 2004);    Department of Defense, DOD Modeling and Simulation (M&S)
Verification, Validation, Accreditation (VV&A), Instruction 5000.61  (Washington, D.C.: May 2003);    Department of Defense, Data Collection, Development, and Management  in Support of Strategic Analysis, Directive 8260.1 (Washington, D.C.:   December 2, 2003); and    Department of Defense, Implementation of Data Collection, Development,  and Management for Strategic Analyses, Instruction 8260.2 (Washington,  D.C.: January 21, 2003).
During the process of identifying generally accepted research standards  we noted that not all studies are conducted the same way. For example,  while all studies use data, not all use baseline data. Likewise, all studies  require analyses, but not all use models or simulation to conduct analyses.  We tailored the research standards we identified as relevant to the MCS, as  shown in table 1.
We used these relevant standards as our criteria to assess the reported  MCS results. All eight key areas of the study process were considered to  have equal importance relative to the soundness and completeness of the  study; that is, a sufficiently serious concern in any category could raise  questions concerning the adequacy and completeness of the report. The  analysts independently reviewed evidence relevant to each subquestion,  including the study itself, the study Terms of Reference, and its strategic  planning guidance. For each of the subquestions in the key study process  areas, the analysts determined whether (1) the evidence had no limitations  or raised no concerns, (2) the evidence had some limitations or raised  some concerns, (3) the evidence had significant limitations or raised  significant concerns, or (4) we could not determine the extent of  limitations or concerns because there was not sufficient information. The  analysts then met, compared, and discussed their individual assessments,  and reached an overall assessment for each subquestion. Areas of the  study where we identified either “some” or “significant” limitations or  concerns were considered to affect the adequacy or completeness of the  study. Additionally, areas of the study that could not be assessed because  of the lack of supporting documentation were considered to affect the  credibility of the study.
Throughout our review PA&E officials told us that the documentation  needed to support and verify the key analytical and decision-making  processes used to conduct the MCS, documentation that was vetted and  approved by DOD leadership and all of the study participants, would not  be completed and available for our review until the study report was  issued. However, after the report was issued, we were told that the report  provides all of the supporting documentation needed and that the other  documentation we requested could not be provided. As a result, we were  unable to determine the adequacy and completeness of the analytical and  decision-making processes that supported the MCS effort to evaluate the  credibility of the study. We believe these processes are significant to the  credibility of the study and its results.
We conducted our review from July 2004 through July 2006 in accordance  with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
subsections: section_title: GAO’s Responses to DOD’s Technical Comments
paragraphs: 1.  DOD disagreed with our assessment that the modeled year—2012—did  not place as much demand for mobility assets in support of smaller  military operations, such as peacekeeping, as other years. DOD also  stated that we incorrectly focused on the number of operations, not  the level of effort. We disagree. The MCS report (Annex A to Appendix  F) made no distinction between the number of lesser contingencies  and the level of effort. Specifically, the Vignettes for Baseline Security  Posture Analysis did not report the level of effort by year and instead  aggregated the data, in many instances across several modeled years.  Consequently, we compared the number of operations conducted in  the model year. Throughout our review, PA&E officials consistently  told us that the completed MCS report would contain all the  documentation needed to support its analyses. Furthermore, although  demand in the modeled year may exceed previous efforts, the MCS  was chartered to assess the ability of the mobility system to support  the National Military Strategy into the next decade.  The size of the  selected model year in relation to efforts conducted between 1941 and  2006 is not at issue. As our report makes clear, our concern is that  modeling what appears to be the least demanding year does not  address whether the United States has sufficient capability to support  national objectives during a peak demand period and may  underestimate and underreport demands to senior decision makers.   2.  DOD disagreed with our observation that the MCS report does not  provide a clear understanding of the direct relationship of warfighting  objectives to transportation capabilities. We disagree. We understand  that achieving a combat task requires delivering the right commodity  to the right place at the right time. However, the specific combat tasks  (e.g., attaining air superiority) necessary to satisfy the commander’s  campaign objectives are not a direct measure of mobility capability.  For example, the problems in using a single metric are reflected in the  MCS Appendix H, where the MCS report states that “the study itself  still had difficulty in evaluating the operational impact of the delivery  of theater support elements,” adding that “we  were unable to  develop a satisfactory mechanism to capture the linkage of the closely  related, but delayed, follow-on support needed...” Finally, the MCS  concludes that “there was no way to model a decrease in   squadron effectiveness if this support was late. Additional effort is  required to develop a methodology for evaluation the operational  impact of support equipment availability.” We continue to believe, and  DOD agreed with our recommendation, that warfighting metrics, in  conjunction with mobility metrics, can give decision makers a full  picture of the capabilities needed to meet a specific warfighting goal.  3.  DOD disagreed with our statement that the MCS was unable to model  the flexible deterrent options/deployment order process and that the  study in fact analyzed flexible deterrent option (FDO) movements to  theater. We do not dispute that DOD analyzed FDO movements as part  of the MCS analysis. However, the degree to which the MCS analyses  successfully modeled FDOs is in question. The MCS report, Appendix  H, stated that an individual FDO is essentially the same as a  deployment order. It also states in that section that “Deployment  orders  are issued to deploy specific capabilities as  commitment decisions are made. This was not modeled due to lack of  data on how DEPORDS would have been issued for an MCO [major  combat operation] deployment.”  In the same paragraph, the MCS  concludes that “the impact on the mobility system of the DEPORD  process should be assessed in follow-on MCS analyses,” adding that  “there is a data deficit on how to model and execute a DEPORD  process.” Furthermore, the MCS report states that “additional analysis  is required to investigate the implications of the DEPORD process  decisions and provide data for future decision-makers to develop a  DEPORD execution process.”   4.  DOD believes our statement concerning homeland defense missions is  misleading and is not sure what the report means by “actual homeland  defense demands.” We removed the word “actual” and clarified our  report to discuss “demands derived from a well defined and approved  concept of operations for homeland defense”, which were not  available for the study according to the MCS report. Furthermore, in  chapter 4, the MCS report states that “maintaining a dedicated  capability to support multiple, nearly simultaneous homeland  defense/civil support events concurrent with the peak demand period  of two overlapping warfights, greatly exceeds programmed lift  capabilities”. This raises questions about the conclusions of the MCS  that there are adequate mobility capabilities to meet national security  objectives. Also, in Chapter 3 of the MCS report, it states that the DOD  homeland defense concept of operations required refinement and was  one of nine issue areas cited within the homeland defense portion of  the study that “need to be addressed and investigated in more detail.”  All of these nine areas potentially impact mobility support for  homeland defense operations. Moreover, the MCS Executive Summary  notes that reassessment of these missions is required as DOD’s role in  homeland defense evolves. The MCS report, chapter 4, concludes by  calling for further refinement of mission requirements, continuing risk  assessments, and an effort to determine corresponding mobility  solutions. We continue to believe that the MCS conclusion that  adequate mobility capability exists is conditional given the results of  the homeland defense portion of the study and that the accuracy and  completeness of the data, modeling, and results for this portion of the  MCS remain in question.   5.  We disagree with DOD’s characterization that our information was  “misleading” regarding the adequacy of some aspects of the MCS’  modeling and data. Furthermore, we continue to disagree with DOD’s  statement that the models and data used by the study were sound and  adequate to assess relevant aspects of missions required to support the  National Military Strategy, and that the results of the study are valid. In  this report, as in our September 14, 2005 report, we reaffirm our  concern that the data and models used by the study may not be sound  and the results may not be valid since the verification, validation, and  accreditation (VV&A) of the models and data used to conduct the  study was not done in accordance with DOD policy or relevant  research standards. VV&A of models and data reduces the risk  inherent in the use of models and simulations by improving the  credibility of modeling and simulation results. We do not dispute  DOD’s assertion that it has relied upon the same models to produce  mobility studies done “since the end of the Cold War”. However, as we  discuss in our report, the MCS report fails to explain or qualify the  impact that identified data or modeling limitations might have on its  results. For example, in the MCS chapter 4, entitled Operational Data,  the MCS states that “data deficiencies negatively affected MCS’s ability  to use current execution data to project future requirements and  assess system performance.” Unclear is the extent to which these  deficiencies impacted the MCS ability to meet the objective of  identifying mobility capability gaps, overlaps, or excesses and provide  associated risk assessments, an MCS objective. Similarly, in the  section of chapter 4 entitled Analysis Tools, the report states that  “MCS analysis revealed several deficiencies in existing mobility  models.” The section concludes with five recommended tool  enhancements but it does not explain the impact that the absence of  these enhanced tools may have. We continue to believe that because of  these modeling and data limitations, the MCS may have incorrectly  estimated the future mobility requirements needed to support  homeland defense missions, smaller contingencies, and major combat  operations.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: Ann Borseth, Assistant Director; Brian Lepore, Assistant Director;  Nabajyoti Barkakati; Renee Brown; Claudia Dickey; Ron La Due Lake;  Oscar Mardis; Deborah Owolabi; Kenneth Patton; and Stephen Woods  made significant contributions to this report.