GAO-08-704R Defense Transportation 

United States Government Accountability Office

Washington, DC 20548 

April 28, 2008 

The Honorable Robert M. Gates 

The Secretary of Defense 

General Larry D. Welch, USAF (Ret.) 

President and CEO 

Institute for Defense Analyses 

Subject: Defense Transportation: DOD Should Ensure that the Final Size and Mix 

of Airlift Force Study Plan Includes Sufficient Detail to Meet the Terms of the Law 

and Inform Decision Makers 

Global mobility1

 is a key component of U.S. national security. Since the end of the 

Cold War, senior decision makers have relied upon Department of Defense (DOD) 

mobility studies to provide insights they need to build and maintain the right mix of 

mobility capabilities. The most recent study, the Mobility Capabilities Study,2

identified the mobility support needed for the full range of strategic operations in the 

context of the September 11, 2001 attacks, the global war on terror, and DOD’s 

evolving global defense posture, all in support of the National Military Strategy. 

According to DOD officials, the department plans to issue the next mobility study—

the Mobility Capabilities Requirements Study—in the spring of 2009. The 2005 

mobility study also assessed requirements for two overlapping war fights, DOD 

support to homeland defense, civil support, lesser contingency operations, 

sustainment of forward-deployed forces, and national strategic missions. In 

accomplishing these missions, DOD depends on its airlift force. 

 

1

The Air Force defines global mobility as the ability to rapidly establish an air-bridge and move military 

capability in support of operations anywhere in the world under any conditions. 2

The intent of the December 2005 Mobility Capabilities Study (MCS) was to identify and quantify the 

mobility capabilities needed to support U.S. strategic objectives into the next decade. The MCS 

determined that the projected mobility capabilities are adequate to achieve U.S. objectives with an 

acceptable level of risk during the period from fiscal years 2007 through 2013; that is, the existing U.S. 

inventory of aircraft, ships, prepositioned assets, and other capabilities were concluded to be 

sufficient, in conjunction with host nation support. The MCS emphasized that continued investment in 

the mobility system, in line with current departmental priorities and planned spending, is required to 

maintain these capabilities in the future. This included, for example, fully funding Army prepositioned 

assets as planned and completing a planned reengineering of the C-5 aircraft. The MCS report also 

made recommendations to conduct further studies, develop plans and strategies, and improve data 

collection and mobility models. 
The National Defense Authorization Act for Fiscal Year 20083

 (hereafter referred to 

as the Act) mandated a requirements-based study on alternatives for the proper size 

and mix of the airlift force to meet the needs of the National Military Strategy to be 

done by a Federally Funded Research and Development Center (FFRDC).4

 The Act 

specifically defined what the study plan should include and set time frames for the 

completion of various events. The FFRDC was to submit a study plan to the 

appropriate congressional committees, the Secretary of Defense, and the Comptroller 

General 60 days after the enactment of the Act. The Act required us to review the 

study plan to determine if it is complete and objective and whether it has any flaws or 

weaknesses in scope or methodology and report to the Secretary of Defense and the 

FFRDC within 30 days. It also required us to include in the report any 

recommendations that the Comptroller General considers appropriate for 

improvements to the study plan. DOD selected the Institute for Defense Analyses 

(IDA) to accomplish the study and signed a task order with IDA outlining the study 

framework. On March 28, 2008, IDA delivered the draft Size and Mix of Airlift Force 

Study Plan to DOD, congressional committees, and us. The draft study plan 

comprises 34 pages of bulleted information, graphs, and diagrams. The seven major 

sections are introduction, background, scope, objective, study management, staged 

approach, and schedule. The single objective of the study is to address the numerous 

airlift issues identified in the Act and to report to the Secretary of Defense and to the 

Congress by January 10, 2009. We assessed the draft Size and Mix of Airlift Force 

Study Plan that IDA delivered on March 28, 2008, for completeness, but we were 

unable to evaluate objectivity or identify flaws or weaknesses in scope or 

methodology.5

 We also commented on another ongoing airlift-related study, the 

Mobility Capabilities Requirements Study, because it is related to the scope of the 

draft Size and Mix of Airlift Force Study Plan. 

DOD uses studies to inform decision making, and study plans are important because 

they define what will be accomplished, what methodologies and assumptions will be 

used in a study, and how it will be done. DOD has used detailed study plans to 

address large and complex issues, including analyses of alternatives supporting 

 3

Pub. L. No. 110-181, §1046 (2008). 4

The Federal Acquisition Regulation sets forth federal policy regarding the establishment and use of 

Federally Funded Research and Development Centers. An FFRDC meets some special long-term 

research or development need which cannot be met as effectively by existing in-house or contractor 

resources. FFRDCs enable agencies to use private sector resources to accomplish tasks that are

integral to the mission and operation of the sponsoring agency. The FFRDC is required to conduct its 

business in a manner befitting its special relationship with the government, to operate in the public 

interest with objectivity and independence, to be free from organizational conflicts of interest, and to 

provide full disclosure of its affairs to the sponsoring agency. FFRDCs are operated, managed, and/or 

administered by either a university or consortium of universities, another not-for-profit or nonprofit 

organization, or an industrial firm, as an autonomous organization or as an identifiable separate 

operating unit of a parent organization. 5

See GAO, Government Auditing Standards: July 2007 Revision, GAO-07-731G (Washington, D.C.: 

July 2007). Government auditing standards define scope as the boundaries of a study that are directly

tied to the study objectives. The scope defines the subject matter that the executors will assess. 

Government auditing standards define methodology as describing the nature and extent of procedures

for gathering and analyzing evidence to address study objectives. Methodology is to obtain sufficient,

appropriate evidence to address objectives and provide reasonable assurance that the evidence is 

sufficient and appropriate to support study findings and conclusions. Methodology includes both the 

nature and extent of procedures used to address objectives. 

Page 2 GAO-08-704R Defense Transportation
acquisitions and execution plans for force management initiatives. Although DOD has 

not published departmentwide guidance, the Army, Air Force, and Navy have 

publications that describe study plans and/or note the use and importance of studies 

in a variety of efforts.6

 Service publications describe a number of detailed elements 

that could be considered for use in study plans and that characterize successful

studies . Our prior work shows that a detailed study plan is a critical part of a wellexecuted study, but may not guarantee a fully successful study. 

To conduct our evaluation, we reviewed IDA’s submission to determine if it fulfilled 

the study plan elements in the Act. We also assessed the completeness of the study 

plan by reviewing the plan and comparing it with study plan descriptions in Army, Air 

Force, and Navy publications that identified the key elements that could be included 

in a study plan. We reviewed prior GAO work that discussed study plans. To obtain 

information on the IDA draft study plan, we interviewed officials from IDA; officials 

from the Office of the Under Secretary of Defense for Acquisition, Technology and 

Logistics; and U.S. Transportation Command. We conducted this performance audit 

from March to April 2008 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that the evidence obtained 

provides a reasonable basis for our findings and conclusions based on our audit 

objectives. 

Results In Brief 

The draft Size and Mix of Airlift Force Study Plan does not meet the terms of the 

Act and lacks sufficient detail for assessment. We are unable to fully assess the draft 

Size and Mix of Airlift Force Study Plan as required in section 1046 of the Act 

because the plan does not contain sufficient detail for us to evaluate its objectivity 

and its scope and methodology. Because the draft study plan did not address all of 

the specified elements in the Act, it is not complete. The draft plan did not include 

specific and explicit references that can be traced directly to the Act, such as the 

assumptions to be included in the study plan and assessments to be accomplished. 

This absence of detail also precludes us from evaluating the scope and methodology. 

Moreover, the plan lacked key details expected in such plans, such as assumptions 

and measures of effectiveness. The lack of details discussed above precludes us from 

making any recommendations concerning improvements to the study plan. DOD 

officials stated that because DOD selected IDA and issued a task order for the study 

only shortly before the mandated deadline, sufficient time was not available to 

produce a more detailed study plan. Nevertheless, DOD is responsible for ensuring 

the statutorily required elements of the study plan are fulfilled. IDA officials told us 

that IDA plans to submit to the Secretary of Defense in June 2008 a final Size and 

Mix of Airlift Force Study Plan that will be more robust. In addition to the 

 6

Studies that might be guided by a study plan include but are not limited to cost, benefit, or

effectiveness analysis of concepts, plans, training, tactics, forces, systems, policies, personnel

management methods, and policies or programs; cost and operational effectiveness analyses (COEA); 

evaluations of force capabilities, organizational structure, administrative policies, procedures, 

methods, systems, and distribution of functions; research and development of data bases, models, and 

methodologies for accomplishing specific studies and analyses; analyses of materiel, personnel, 

logistics, and management systems; and studies to establish materiel requirements. 

Page 3 GAO-08-704R Defense Transportation
independent draft Size and Mix of Airlift Force Study Plan, we note that DOD is 

conducting another study that may also inform decision makers on airlift issues. 

Accordingly, we are recommending that the Secretary of Defense direct the Under 

Secretary of Defense for Acquisition, Technology and Logistics to ensure that the 

final Size and Mix of Airlift Force Study Plan include (1) sufficient detail to address, 

at a minimum, elements mandated in section 1046 of the National Defense 

Authorization Act for Fiscal Year 2008; and (2) sufficient detail to inform decision 

makers on airlift issues. 

In oral comments on a draft of this report, DOD concurred with our recommendation. 

DOD’s comments are discussed in more detail at the end of this report. DOD also 

provided technical comments and we have incorporated them where appropriate. 

Background 

In previous GAO work, we noted the importance of study plans and usage of study 

plans as a best practice.7

 Specifically, we found that governmental agencies and the 

private sector rely on detailed study plans, or data collection and analysis plans, to 

guide the development of studies and the collection and analysis of data. 

Additionally, we stated that a study plan provides a feedback loop that links the 

outcomes of the study and subsequent analysis to the original goals and objectives of 

the study. We also found that particularly large and complex issues may benefit from 

a study plan. GAO also identified a best practice for a study plan process that mirrors 

information found in service publications concerning study plans. 

Service publications describe the role of studies in complex decision making in areas 

such as acquisitions, comparing alternatives, and evaluations of force capabilities. 

Service publications also describe elements, such as assumptions and measures of 

effectiveness (MOEs), that may be considered for inclusion in a study plan. The Navy 

publication that discusses studies describes determining assumptions, statements 

related to the study that are taken as true in the absence of facts, as a key step in 

study plan development.8

 The Army publication that discusses preparation of a study 

also lists assumptions as a part of a study plan.9

 The Air Force analysis handbook 

contains a study plan outline that includes assumptions as well.10 According to these 

service publications, MOEs are the level of success to be achieved or how well tasks 

are performed. All of the service publications refer to MOEs as part of a study plan. In 

the Army and Navy publications, selection of the MOEs is described as perhaps the 

most crucial part of any analysis. 

 7

See GAO, Military Readiness: Navy’s Fleet Response Plan Would Benefit from a Comprehensive 

Management Approach and Rigorous Testing, GAO-06-84 (Washington, D.C.: Nov. 22, 2005). 8

Navy Warfare Development Command, Study Planning and Conduct Guide, Mr. Richard C. Rigazio, 

Operations Research Analyst (August 2007). 9

Department of the Army Pamphlet 5–5, Guidance for Army Study Sponsors, Sponsor’s Study 

Directors, Study Advisory Groups, and Contracting Officer Representatives, Headquarters 

(Washington, D.C. Nov 1, 1996). 10Office of Aerospace Studies, Analysis Handbook, A Guide for Performing Analysis Studies: For 

Analysis of Alternatives or Functional Solution Analyses (July 2004). 

Page 4 GAO-08-704R Defense Transportation
The Draft Size and Mix of Airlift Force Study Plan Does Not Meet the Terms 

of the Act and Cannot Be Fully Assessed 

We are unable to fully assess the draft Size and Mix of Airlift Force Study Plan as 

required in section 1046 of the Act because it does not contain sufficient detail for us 

to evaluate the objectivity or to evaluate the scope and methodology. Since the draft 

study plan did not address all of the specified elements in the Act, it is not complete. 

This absence of detail also precludes us from evaluating the scope and methodology. 

In the absence of a complete study plan, the final study may not sufficiently inform 

decision makers concerning the alternatives for the size and mix of the airlift force to 

meet requirements of the National Military Strategy. 

The draft study plan is not complete because it did not address all of the required 

elements in the Act. While IDA officials described the draft as generally addressing 

the three main elements cited in the Act, we found that it lacks specific and explicit 

references that can be traced directly to the Act. Specifically, the Act identified 11 

assumptions, at a minimum, to be included in the study plan.11 For example, one 

required assumption involved the new capability in airlift to be provided by the KC(X) 

tanker aircraft. The draft study plan included a 5-bullet slide concerning the 

assessment of the KC(X) as an airlifter, but did not include the required assumptions. 

Additionally, the Act required the study plan to include assumptions concerning airlift 

mobility requirements in support of homeland defense and national emergencies. 

However, the plan only included a number of bulleted statements about the homeland 

defense and national emergencies missions and contained no assumptions. Without 

the required assumptions, the draft Size and Mix of Airlift Force Study Plan does 

not satisfy the Act and is not complete. 

The draft study plan also does not contain key details sufficient for us to evaluate the 

scope and methodology. Service publications reflect the importance of key steps 

such as identification of MOEs. According to the service publications, perhaps the 

most crucial part of any analysis is the selection of appropriate measures of 

effectiveness, which are central to evaluating alternatives. They enable decision 

makers to compare the results in a study.12 For example, MOEs could allow decision 

makers to compare the results of two analyses that measure different airlift force 

mixes. In the draft study plan, mixes of two aircraft (C27 and C-130) are to be 

compared for their fleet effectiveness. In such an example, the time required for an 

aircraft fleet to accomplish a mission, such as moving cargo or people, can be 

measured in terms of time required, tons moved, or miles flown. The task order for 

the draft Size and Mix of Airlift Force Study Plan specified the use of MOEs. The 

draft Size and Mix of Airlift Force Study Plan includes a slide concerning 

effectiveness analyses but, for the purposes of our analysis, it does not detail the 

MOEs or how they are to be used. Without critical details such as MOEs, the final 

study may not sufficiently inform decision makers concerning airlift issues. 

 11Pub. L. No. 110-181, §1046 (d)(1) (2008). 12According to the Air Force analysis handbook, MOEs are important to a warfighter because they 

express both system worth (ability to contribute to a warfighter’s immediate goal) and military worth 

(ability to contribute to high-level goals of winning the war). 

Page 5 GAO-08-704R Defense Transportation
In our discussion with IDA officials concerning the draft study plan, they 

acknowledged the importance of the use of assumptions and MOEs in study plans. 

DOD officials stated that DOD selected IDA only shortly before the mandated 

deadline for a draft study plan and added that sufficient time was not available to 

produce a more detailed draft study plan. The plan states that its final stage (phase) 

will ensure all issues raised in the Act are addressed. In our discussion with IDA 

officials, they explained that the March 2008 version of the Size and Mix of Airlift 

Force Study Plan is a draft and that the final study plan, scheduled for completion in 

June 2008, will be more robust and detailed. We note that while DOD is using IDA to 

perform the study plan pursuant to a task order, DOD is responsible for ensuring that 

the statutorily required elements of the study plan are fulfilled. We also note that, 

pursuant to the Act, we were to provide any recommendations that the Comptroller 

General considered appropriate for improvement to the study plan. The lack of 

details discussed above precludes us from making any recommendations concerning 

improvements to the study plan. 

In the Navy study guide publication, one of the first steps in initiating a study is 

identifying potential uses for anticipated study results. While our scope only included 

the draft Size and Mix of Airlift Force Study Plan, we note that DOD plans to publish 

results from both the Size and Mix of Airlift Force Study and the Mobility Capabilities 

Requirements Study in 2009 as decision makers consider airlift requirements. We 

believe that it is possible that decision makers may compare the findings and 

assumptions of the Size and Mix of Airlift Force Study with the airlift portion of the 

Mobility Capabilities Requirements Study. If the Mobility Capabilities Requirements 

Study does not consider the objectives and assumptions of the Size and Mix of Airlift 

Force Study, it is unclear as to whether the results of the two studies can be 

compared and therefore the study results may not be fully useful to decision makers. 

Conclusion 

Well-executed airlift and mobility studies that inform decision making are important 

as DOD continues its efforts to sustain, modernize, and recapitalize its airlift 

programs. Detailed study plans are a critical part of well-executed studies and should 

include key details such as assumptions and MOEs. Although the draft Size and Mix 

of Airlift Force Study Plan does not fulfill the requirements of the Act, the final 

version may include corrections and refinements. 

Recommendations for Executive Action 

We recommend that the Secretary of Defense direct the Under Secretary of Defense 

for Acquisition, Technology and Logistics to ensure that the final Size and Mix of 

Airlift Force Study Plan includes sufficient detail to address, at a minimum, 

elements mandated in section 1046 of the National Defense Authorization Act for 

Fiscal Year 2008 and ensure that the study plan includes sufficient detail to inform 

decision makers on airlift issues. 

Page 6 GAO-08-704R Defense Transportation
Agency Comments and Our Evaluation 

In oral comments on a draft of this report, DOD concurred with our recommendation 

that DOD ensure that the final study plan include sufficient detail to address, at a 

minimum, elements mandated in law and sufficient detail to inform decision makers 

on airlift issues. As part of the comments, an official from the Office of the Under 

Secretary of Defense for Acquisition, Technology and Logistics reiterated that IDA 

was not given adequate time to create a detailed study plan in compliance with 

established standards. DOD stated the intention to continue monitoring IDA’s work 

on the final study plan, adding that DOD fully expects the final study plan to comply 

with established DOD standards regarding studies. DOD also provided technical 

comments and we have incorporated them where appropriate. 

________ 

We are sending copies of this report to the Senate and House Appropriations and 

Armed Services Committees and other interested congressional committees; the 

Under Secretary of Defense for Acquisition, Technology and Logistics; the 

Commander of U.S. Transportation Command; and the Director, Office of 

Management and Budget. In addition, this report will be available at no charge on the 

GAO Web site at http://www.gao.gov. 

If you have any questions regarding this report, please contact me at (202) 512-8365 

or solisw@gao.gov. Contact points for our Offices of Congressional Relations and 

Public Affairs may be found on the last page of this report. GAO staff that made

contributions to this report include Ann Borseth, Ron La Due Lake, Charles Perdue, 

Karen Thornton, Karen Werner, and Steve Woods. 

William M. Solis, Director 

Defense Capabilities and Management 

Page 7 GAO-08-704R Defense Transportation
Related GAO Products

Defense Acquisition: KC-135 Recapitalization Analysis of Alternatives Does Not 

Inform Decision Makers Regarding Cost, Effectiveness, and Suitability. 

GAO-08-69CR. Washington, D.C.: January 8, 2008. 

Defense Acquisitions: Air Force Decision to Include a Passenger and Cargo 

Capability in Its Replacement Refueling Aircraft Was Made without Required 

Analyses. GAO-07-367R. Washington, D.C.: March 6, 2007. 

Defense Transportation: Study Limitations Raise Questions about the Adequacy 

and Completeness of the Mobility Capabilities Study and Report. GAO-06-938. 

Washington, D.C.: September 20, 2006. 

Military Readiness: Navy’s Fleet Response Plan Would Benefit from a 

Comprehensive Management Approach and Rigorous Testing. GAO-06-84. 

Washington, D.C.: November 22, 2005. 

(351189) 

Page 8 GAO-08-704R Defense Transportation
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, DC 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

PRINTED ON RECYCLED PAPER