DOD Manufacturing Arsenals: Actions Needed to Identify and Sustain Critical Capabilities
Released: 2015-11-17T00:00:00

Letter

Background

Role and Organizational Structure of Manufacturing Arsenals

The “defense industrial base” includes all commercial and government- owned facilities that are responsible for the design, production, delivery, and maintenance of military weapon systems, subsystems, and components or parts that fulfill U.S. military requirements. The portion of the defense industrial base that is assigned to and forms an essential part of DOD’s organization is referred to as the “organic defense industrial base.” The organic defense industrial base consists of resource providers, acquisition and sustainment planners, and manufacturing and maintenance performers, such as DOD’s government-owned manufacturing arsenals and maintenance depots. These government- owned and -operated installations, including the three manufacturing arsenals, provide services for a variety of customers, including the Army, the Navy, the Air Force, and some non-DOD agencies and foreign countries. The Army’s Industrial Operations activity group, a subset of the organic defense industrial base, includes the Army’s manufacturing arsenals, maintenance depots, ammunition plants, and storage sites. According to the Army’s Organic Industrial Base Strategic Plan 2012- 2022, workforces and infrastructures of the Army Industrial Operations activity group are to be sized and adjusted accordingly over time to sustain critical manufacturing and core depot capabilities to support war- fighting equipment during current and future contingency operations. Within the Army, the Office of the ASA (ALT) serves in an oversight capacity and is responsible for establishing the policy and goals for the Army’s industrial base program.

The three manufacturing arsenals are operated by the Army, managed by AMC, and under the direct command and control of the Army’s Life Cycle Management Commands. Each manufacturing arsenal is aligned with a command that oversees the kind of work done at that arsenal. Specifically, Rock Island and Watervliet Arsenals are aligned with the TACOM Life Cycle Management Command and its mission of developing, acquiring, fielding, and sustaining ground systems. The work performed at Pine Bluff Arsenal is aligned with the Joint Munitions Command Life Cycle Management Command, the logistics integrator for life-cycle management of ammunition.

While the manufacturing arsenals are under the Army’s operational control, organizations within the Office of the Secretary of Defense (OSD) perform policy, planning, program, and resource management functions for the industrial base, which includes the arsenals. Within OSD, the Office of the Assistant Secretary of Defense for Logistics and Materiel Readiness serves as the principal logistics official within senior DOD management. This office prescribes policies and procedures for the conduct of logistics, maintenance, materiel readiness, strategic mobility, and sustainment support within DOD. For example, the Office of the Deputy Assistant Secretary of Defense for Maintenance Policy and Programs, under the authority of the Office of the Assistant Secretary of Defense for Logistics and Materiel Readiness, held primary responsibility for consolidating and submitting DOD’s September 2014 report to Congress. Additionally, the Defense Logistics Agency (DLA) is DOD’s logistics combat support agency, whose primary role is to provide supplies and services to America’s military forces and sometimes procures items from the manufacturing arsenals. Figure 1 shows the structure of DOD’s manufacturing arsenal organization, including relevant DOD entities.

Each manufacturing arsenal has been designated by the Secretary of the Army as a Center of Industrial and Technical Excellence. This designation provides authority under section 2474 of Title 10 U.S. Code to partner with and lease facilities to industry on programs relating to core maintenance and technical expertise.

Pine Bluff Arsenal is designated as a Center of Industrial and Technical Excellence for chemical and biological defense equipment.

Rock Island Arsenal Joint Manufacturing and Technology Center is designated as a Center of Industrial and Technical Excellence for mobile maintenance systems; foundry operations; and add-on armor design, development, and prototype fabrication.

Watervliet Arsenal Joint Manufacturing and Technology Center is designated as a Center of Industrial and Technical Excellence for manufacturing cannon and mortar systems.

Figure 2 shows the manufacturing arsenals’ geographic locations within the United States and their missions.

Legislative Authorities That Affect the Workload Process

The manufacturing arsenals are subject to various legislative provisions that affect the work they do and how this work is allocated. These include section 4532 of Title 10 U.S. Code, commonly referred to as the Army Arsenal Act, and other statutes that authorize the establishment of public-private partnerships, including direct sales, research and development, and facilities use agreements, any of which may affect how much work arsenals undertake. The Army Arsenal Act specifically requires the Army to have its supplies made in U.S. government factories or manufacturing arsenals, provided that they can produce the supplies on an economical basis. The economical basis determination, according to the Army Arsenal Act’s implementing guidance, is based on a comparison of the government’s manufacturing costs with the cost of purchasing the supplies commercially—a process commonly referred to as the “make-or-buy” analysis. More broadly, section 2535 of Title 10 U.S. Code, which applies to all the armed forces, declares that it is Congress’s intended policy that “to the maximum extent practicable, reliance will be placed on private industry for support of defense production.” There are also a number of authorities that the Army’s manufacturing arsenals may use to establish commercial-like relationships with other entities. For example, section 2474 of Title 10 U.S. Code authorizes the establishment of public-private partnerships between Army manufacturing arsenals and private entities. Additionally, section 4544 of Title 10 U.S. Code is an example of a direct sales statute, which gives the manufacturing arsenals the authority to enter into cooperative agreements—such as sales and leasing contracts—with non-Army entities, both public and private.

Funding Sources for the Manufacturing Arsenals and How Customers’ Rates Are Set

Funding for the manufacturing arsenals is managed through the Army Working Capital Fund. Section 2208 of Title 10 U.S. Code authorizes the Secretary of Defense to establish Working Capital Funds to finance inventories of supplies and industrial activities that provide common services such as repair, manufacturing, or remanufacturing. When a private entity, such as the prime integrator of a major weapon system, or a government component or agency places an order with a manufacturing arsenal for equipment or services, payments are made to the Working Capital Fund on a reimbursable basis. Charges for goods and services provided through the fund include the full costs of the goods and services provided and amounts set for the depreciation of capital assets. According to DOD financial reports, in fiscal year 2014, the combined total operating expenses incurred at the three manufacturing arsenals— which were applied to the Army Working Capital Fund—were approximately $400 million.

The Army’s Industrial Operations activity group sets the rates customers will pay the manufacturing arsenals for equipment or services they order, on a direct labor-hour basis. The process for setting rates begins 18 months before the start of the fiscal year in which the manufacturing will be performed. Based on the anticipated level of future work, the managers of the manufacturing arsenals propose hourly rates that will allow them to recover all of their operating costs. Then, the proposed business plan and rates and the manufacturing arsenal budget are approved through the Army chain of command. Ultimately, the rates are set by the Department of the Army and DOD. Although the goal of the Army Working Capital Fund—unlike the goal of a profit-oriented commercial business—is to be self-supporting by recovering only the cost of supplies, services performed, and applicable administrative expenses, a manufacturing arsenal may end the year with more or less resources than it had originally anticipated—depending on whether or not its actual costs and workload over the fiscal year were as forecasted. In such cases, there may be a rate increase in a subsequent year in an effort to offset the losses of a prior year or a rate reduction to offset gains.

Additionally, every year the Army is required to include, in annual budget documents submitted to Congress to support the President’s fiscal year budget request, an estimate of funds required in that fiscal year to cover the costs of unutilized or underutilized plant capacity at Army arsenals. This funding is referred to as Industrial Mobilization Capacity funding. However, Congress may or may not appropriate funds specifically for this purpose from one year to the next. According to data provided by DOD, two of the manufacturing arsenals—Pine Bluff and Watervliet—received funding through the Industrial Mobilization Capacity account, a subaccount of the Army’s Working Capital Fund, every fiscal year from 2000 through 2006—and Rock Island Arsenal received funds from fiscal year 2001 through 2007—to cover the costs of unutilized or underutilized plant capacity. In fiscal years 2008 through 2013, the manufacturing arsenals did not receive funding for Industrial Mobilization Capacity, although appropriations specifically for this purpose were again made in fiscal year 2014 in the amount of $150 million. Additionally, in fiscal year 2015, Congress appropriated $225 million to the Working Capital Fund for maintaining competitive rates at arsenals, a purpose which is distinct from covering costs of underutilized or unutilized plant capacity.

The levels of work at the manufacturing arsenals have changed over time. Specifically, according to the Secretary of the Army, there has been a precipitous drop in demand for Army materiel that has resulted in a decline in workload and an increase in overhead rates at the manufacturing arsenals. Army personnel data for these arsenals indicates that, during fiscal years 2000 through 2002 (prior to the start of operations in Iraq), combined total workload at the three manufacturing arsenals ranged from approximately 1.5 million to 1.7 million direct labor hours each year. During fiscal years 2003 through 2012 (during operations in Iraq and Afghanistan), the combined total workload at the three manufacturing arsenals each year ranged from approximately 1.6 million to 3.0 million direct labor hours. In fiscal years 2013 and 2014 (after operations in Iraq had ended), the combined total workload each year ranged from approximately 1.2 million to 1.7 million direct labor hours.

Because the manufacturing arsenals operate under the Army Working Capital Fund, they must include all costs of running the installation—such as costs for security and facility maintenance—in the rates they charge customers. As a result, when the volume of work decreases, as it did in fiscal years 2013 and 2014, those fixed costs of operation must be spread over a shrinking base of work; the result is an increased cost per unit and higher rates. For example, according to a briefing developed by AMC and presented to Congress in July 2014, during fiscal years 2013 and 2014, the stabilized rate at Pine Bluff increased from approximately $126 per hour to approximately $135 per hour, which represented approximately a 7 percent increase. In the same time frame, the stabilized rate at Rock Island increased from approximately $112 per hour to approximately $137 per hour—approximately a 23 percent increase. Also during this time frame, the stabilized rate at Watervliet increased from approximately $195 per hour to approximately $202 per hour— approximately a 4 percent increase. According to AMC’s briefing, in July 2014 the Army projected that stabilized rates for fiscal year 2015 would have been significantly higher without the infusion of Arsenal Sustainment Initiative funds to attempt to make arsenal rates more competitive. For example, the Army projected that the stabilized rate for Rock Island would increase to approximately $285 per hour, while the stabilized rate at Watervliet was projected to increase to approximately $400 per hour— approximately double Watervliet’s rate for fiscal year 2014. Officials at the manufacturing arsenals told us that even though they actively market the arsenals’ capabilities to DOD program officials, as their rates increase, the arsenals lose even more customers and workload. These officials explained that this results in a continuing cycle of decreasing requirements and increasing rates, a pattern which several DOD officials we interviewed referred to as a “death spiral.”

DOD Has Taken Actions over the Last 3 Years to Assign Work to the Manufacturing Arsenals, but These Actions Have Not Generated Sufficient Revenue to Recover Operating Expenses

Over the past 3 years, DOD has taken various actions to assign work to the manufacturing arsenals, but these actions have not generated sufficient revenue to recover their operating expenses. Moreover, we found that DOD may not always appropriately consider the manufacturing arsenals as a source of manufacture in a given situation, because it does not have clear, step-by-step implementing guidance on how to conduct make-or-buy analyses to determine whether to procure an item from the arsenals or the private sector. In response to the manufacturing arsenals’ inability to generate sufficient revenue to recover their operating expenses, Congress appropriated funds in fiscal years 2014 and 2015 to help recover the arsenals’ operating expenses and allow them to maintain competitive rates.

DOD Has Taken Actions to Assign Work to the Arsenals

Since 2012, DOD has taken various actions to assign work to the manufacturing arsenals. For example, in December 2012 ASA (ALT) issued a memorandum directing Program Executive Officers, Program Managers, and Product Support Managers in the Army’s acquisition community to use a market research tool called the Materiel Enterprise Capabilities Database when conducting research to determine whether an item should be made at a manufacturing arsenal or bought from the private sector. The tool, available to all DOD components, provides access to information on the capabilities that are available at each of the manufacturing arsenals. According to Army officials, the intent of this effort is to make the Army and other DOD components more aware of the manufacturing arsenals’ capabilities, in the hope that such increased awareness will lead these organizations to send the arsenals more work.

Additionally, in an effort to encourage Program Executive Officers and Program Managers to take advantage of the manufacturing arsenals’ capabilities, ASA (ALT) issued a memorandum in May 2013 directing Program Executive Officers to report annually on the work they have provided to the arsenals. The memo also stated that, when possible, decisions to use the manufacturing arsenals, contractor support, or some combination of arsenals and contractor support should occur early in the acquisition process, for example, during market research or when conducting make-or-buy analyses. Subsequently, in April 2014, the Secretary of the Army directed AMC to work directly with DLA to develop a plan and schedule to “make the manufacturing arsenals a DLA source of supply” for Army-related manufacturing requirements. The secretary’s memorandum noted that it was the Army’s position that all Army-related parts in the DLA inventory that the manufacturing arsenals were capable of manufacturing should first be obtained from the arsenals. In June 2014, DLA determined that it had statutory authority to order from the Army’s manufacturing arsenals when procuring supplies for the Army, but noted that the Army would still need to conduct make-or-buy analyses and provide DLA with a list of items that must be manufactured at the arsenals. In July 2014, AMC requested that DLA “make the manufacturing arsenals a primary source of supply” for a list of 133 items, such as mounting plates and brackets, that the arsenals identified as items they had manufactured previously.

In fiscal year 2014—as part of its implementation of section 8141 of the Fiscal Year 2014 Consolidated Appropriations Act directing the Secretary of the Army to assign sufficient work to the manufacturing arsenals to maintain their critical capabilities—ASA (ALT) reviewed Program Executive Officers’ portfolios to identify work that could be directed to the arsenals in order to reduce the likelihood of a rate increase and help to maintain critical capabilities. ASA (ALT) directed that a program management review be conducted every 6 months on the status of the originally identified workload, which included 26 projects—such as the M320 grenade launcher—to be directed to the manufacturing arsenals during fiscal years 2015 and 2016.

Finally, as part of its September 2014 Report on Manufacturing Arsenal Study, DOD stated that it would continue to encourage the arsenals to use public-private partnerships. These public-private partnerships include, for example, commercial tenants who either rent space or provide services in kind to the manufacturing arsenals. The intent of the partnerships is to reduce the arsenals’ overhead, maintenance, and product costs. As of October 2014, AMC had also developed a draft implementation guide for a business development and partnership program to coordinate partnering management, with the objective of helping to positively affect the net operating result of AMC activities.

Actions to Assign Work Have Not Generated Sufficient Revenue, and DOD May Not Always Appropriately Consider the Manufacturing Arsenals as a Source of Manufacture

The various actions that DOD has taken to assign work to the manufacturing arsenals, as described above, have not generated sufficient revenue to recover the arsenals’ operating expenses and do not ensure that DOD is appropriately considering the arsenals as a source of manufacture. While DOD’s efforts to assign work have increased revenue, the increases have been small relative to the manufacturing arsenals’ operating expenses. For example,

DLA officials told us that as of August 2014, DLA had provided approximately $10 million of work to the manufacturing arsenals in fiscal year 2014. This work generated enough revenue to recover approximately 2 percent of the arsenals’ total expenses in that time frame.

AMC’s Assistant Deputy Chief of Staff for Logistics Integration explained that, as of August 2014, the manufacturing arsenals were engaged in 22 public-private partnerships that yielded a total of approximately $11 million in revenue for the arsenals. This income would recover approximately 3 percent of the manufacturing arsenals’ total expenses in fiscal year 2014. Further, multiple officials from OSD, the Army, and the manufacturing arsenals told us that while public-private partnerships are a good source of a small amount of revenue, they are not the long-term solution to the arsenals’ ongoing shortage of work.

In addition, DOD’s actions to assign work to the manufacturing arsenals have not ensured that they are consistently considered as a source of manufacture. Specifically, the Army’s effort to obtain work from DLA involves make-or-buy analyses that are to be conducted to determine whether to purchase an item from DOD’s industrial facilities, such as the manufacturing arsenals, or from the private sector. However, based on our review of relevant documents and interviews with DOD and Army officials, we found that the Army does not have clear, step-by-step implementing guidance—such as an instruction or guidebook—on how to conduct make-or-buy analyses. Army Regulation 700-90 states that ASA (ALT) is responsible for determining where an item should be procured from and directs Program Managers or Program Executive Officers to conduct the analyses that inform these determinations. The Army’s regulation contains broad descriptions on how to conduct make-or-buy analyses. It notes that the cost estimate for making the item at a manufacturing arsenal should include the direct costs and only those indirect costs that would change as a result of changes in the number of items manufactured. Additionally, while the Army has issued a Cost Benefit Analysis Guide that provides guidance on conducting cost benefit analyses, this guide does not include specific, step-by-step information on the policies, responsibilities, or procedures for conducting make-or-buy analyses. According to AMC officials, no prescriptive guidance on how to conduct these analyses has been issued so there is flexibility in how they are conducted.

While the existing regulation may provide flexibility, some manufacturing arsenal officials responsible for conducting these analyses at two of the three arsenals stated that the guidance is not clear and that, as a result, they have requested more detailed, step-by-step guidance to ensure that they conduct these analyses consistently. For example, officials at one manufacturing arsenal told us that in conducting make-or-buy analyses, they are supposed to remove sunk costs when developing their estimates. However, these officials, who are responsible for conducting cost estimates, told us they did not know how to calculate their rates without including the sunk costs. Officials at another manufacturing arsenal told us that the process for conducting make-or-buy analyses is unclear and expressed their opinion that a joint DOD instruction is needed to better implement the process. Federal internal control standards emphasize the importance of establishing detailed policies, procedures, and practices to ensure that such guidance is an integral part of operations. According to Army officials, having clear implementing guidance to help ensure that make-or-buy analyses are consistently conducted would not guarantee that the manufacturing arsenals receive sufficient workload to recover their operating expenses, but it would ensure that they are appropriately considered. In the absence of clear, step-by-step implementing guidance, such as an instruction or guidebook, that outlines how to conduct make-or-buy analyses, DOD cannot provide reasonable assurance that it is appropriately considering the manufacturing arsenals as a potential source of manufacture, thereby potentially limiting the arsenals’ ability to generate revenues.

Congress Appropriated Funds to Help the Manufacturing Arsenals Recover Operating Expenses and Maintain Competitive Rates

Because DOD’s efforts to assign work to the manufacturing arsenals have not generated sufficient revenue, Congress appropriated $150 million to the arsenals in fiscal year 2014 to recover their operating expenses and maintain competitive rates. These amounts were to be made available in the Industrial Mobilization Capacity subaccount on the condition that the Secretary of the Army assign sufficient workload to the arsenals to sustain their critical manufacturing capabilities and ensure cost efficiency, among other goals. DOD analyzed the financial positions, projected rates, and future workloads of the three manufacturing arsenals and allocated the funding based on the relative need of each. AMC determined that, without this funding, the projected losses at Rock Island and Watervliet Arsenals would have placed those installations in a negative financial position by the end of fiscal year 2014 and that these manufacturing arsenals would need to raise their rates substantially to recover their operating losses. Then, in fiscal year 2015, Congress provided $225 million to the Army’s Working Capital Fund to help maintain competitive rates at the manufacturing arsenals. Of the funds Congress appropriated in fiscal years 2014 and 2015, AMC allocated the funds to the arsenals as follows:

Rock Island Arsenal: $110 million in fiscal year 2014 and $135 million in 2015.

Watervliet Arsenal: $30 million in fiscal year 2014 and $80 million in fiscal year 2015.

Pine Bluff Arsenal: $10 million in fiscal year 2014 and $10 million in fiscal year 2015.

According to DOD accounting reports and Army officials, the appropriated funds were directly applied to the manufacturing arsenals’ revenues to offset losses and reduce rates. However, these funds were not accompanied by any work. Officials at each of the three manufacturing arsenals told us that while the funds from Congress were helpful, they would prefer to receive additional work instead, to recover operating expenses, lower rates, and sustain the arsenals’ manufacturing capabilities.

DOD Is Not Strategically Positioned to Sustain the Manufacturing Arsenals’ Critical Capabilities

DOD is not strategically positioned to sustain the manufacturing arsenals’ critical capabilities because, although it has a strategic plan that covers the manufacturing arsenals, it has not identified fundamental elements, such as time frames, necessary to implement this plan and achieve its goals and objectives. Furthermore, because DOD has not established a process for identifying the manufacturing arsenals’ critical capabilities, developed a method for determining a minimum level of workload to sustain these capabilities at each of the arsenals, and identified and implemented this process and method via guidance—such as a DOD instruction—the department is not positioned to determine the minimum workloads or levels of manufacturing equipment and personnel needed to sustain these capabilities.

DOD Has a Strategic Plan for the Manufacturing Arsenals but Has Not Identified Fundamental Elements for Implementation

In 2012, the Army issued its Organic Industrial Base Strategic Plan 2012- 2022 (strategic plan), which identifies several goals and objectives related to the three manufacturing arsenals: Institutionalize Army sustainment functions so that the Army’s priorities inform the manufacturing arsenals’ production schedules.

Assess which competencies and capabilities are essential to the organic industrial base.

Fund 100 percent of the minimum level of work the manufacturing arsenals must have in order to exercise their critical capabilities sufficiently to sustain them.

The strategic plan also outlines the following objectives related to the arsenals: Identify and document critical manufacturing capabilities that the manufacturing arsenals should have.

Adjust equipment and personnel at the manufacturing arsenals to sustain these critical manufacturing capabilities.

Establish an integrated Human Capital Investment Plan that supports current and future capability requirements.

Continue to promote public-private partnerships.

However, the Army has not identified other fundamental elements associated with the achievement of the strategic plan’s goals and objectives. Standard practices for project management call for agencies to conceptualize, define, and document specific goals and objectives in their planning processes and to identify the appropriate steps, milestones, time frames, and resources they need to achieve those goals and objectives. The Army’s strategic plan does not contain any of these fundamental elements. Furthermore, Army officials informed us that there are no other documents, such as an implementation plan, that contain these fundamental elements and that they have no plan to document them. They explained that such documentation is not needed, because they fully understand the efforts being taken to implement this strategy, including hosting summits to share ideas on how to increase workload levels at the manufacturing arsenals, working directly with DLA to make the arsenals a DLA source of manufacture, and collaborating with other countries to potentially increase foreign military sales of arsenal goods and services. However, unless it identifies and documents these fundamental elements by including information that would be useful in determining DOD’s progress toward achieving its stated goals and objectives, the department is not strategically positioned to sustain the manufacturing arsenals’ critical capabilities.

DOD Has Not Identified the Manufacturing Arsenals’ Critical Capabilities

Achievement of the strategic plan’s goals and objectives is predicated on the identification of the manufacturing arsenals’ critical capabilities. However, these critical capabilities have not been identified and documented, as called for in the 2012 strategic plan. Further, DOD has not developed and implemented an agreed-upon process for determining these capabilities, although such efforts are under way. DOD has previously undertaken efforts to identify the manufacturing arsenals’ critical capabilities, but those earlier efforts had shortcomings. For example, in April and May 2013, each of the three manufacturing arsenals submitted studies to AMC that they had conducted to identify their critical capabilities and the minimum workload they would need in order to sustain those capabilities. These studies were completed in response to the Senate Armed Services Committee’s report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2013, which directed the Secretary of Defense to identify critical manufacturing capabilities and capacities that should be government- owned and -operated, as well as the level of work needed to sustain those capabilities. However, AMC officials responsible for overseeing these studies, as well as officials from OSD and the manufacturing arsenals, told us that the arsenals were not given a standardized, consistent method to follow in identifying their critical capabilities and minimum workloads. Rather, AMC officials told us, they relied on the manufacturing arsenals to each develop their own unique method. A senior OSD official described the resulting process as unsound. Each manufacturing arsenal declared what it believed to be its own critical capabilities in an unstructured way and based its analysis on then-current personnel levels. A senior official at ASA (ALT) expressed a similar opinion, saying that the manufacturing arsenals had labeled everything they were doing at that time, including assembly work, as critical. Nonetheless, the results of each of the three manufacturing arsenals’ studies were consolidated into a single report that listed critical capabilities for each of the three arsenals and the estimated workloads necessary to sustain them; this report was submitted to Congress in August 2013.

Recognizing the shortcomings of earlier efforts to identify critical capabilities, ODASD (MPP) commissioned a study in March 2013 to (1) establish a process for identifying critical manufacturing capabilities and (2) develop a method to identify the minimum workloads needed to sustain these capabilities. In mid-August 2015, an ODASD (MPP) official told us that OSD hoped to have this process for identifying critical capabilities completed by December 2015, more than 2 years after the effort had begun. This official explained that the working group focused on this effort did not initially include representation from all of the relevant stakeholders, including representatives from the manufacturing arsenals. The official told us that, as a result, the effort had been temporarily paused so that the working group conducting the study could incorporate the new stakeholders.

According to ODASD (MPP) officials, the process being developed to identify critical capabilities and the method for determining the minimum workload needed to sustain them will form the basis of a DOD instruction applicable to all of the military services. ODASD (MPP) officials explained that they have not begun drafting the instruction, and they could not provide an estimated time frame for when the instruction would be issued. They noted that even once the related study is completed, it could take several years to finalize and issue the instruction. ODASD (MPP) officials stated that they expect the development of the instruction to be challenging, given the divergent views within Army leadership, for example on what critical capabilities are needed at the manufacturing arsenals. Until the department completes the study it commissioned in March 2013 and issues its implementing instruction, DOD will continue to lack an agreed-upon process for identifying the manufacturing arsenals’ critical capabilities and a method for determining a minimally-sustaining workload to sustain those capabilities.

Because DOD has not identified the manufacturing arsenals’ critical capabilities or determined the minimum levels of workload needed to sustain these capabilities, as called for in the strategic plan, DOD is not positioned to achieve the strategic plan’s other goals and objectives. For example, it cannot determine the amount of resources that would be needed to assign 100 percent of the minimum level of work required to sustain the critical capabilities. Further, it is not able to adjust the equipment and personnel levels at the manufacturing arsenals to levels that would sustain these capabilities. Regarding equipment and personnel levels, there have been some efforts to make adjustments to reduce operating expenses. For example, in mid-2014, Watervliet Arsenal assessed its manufacturing equipment and identified several dozen machines that it could potentially lay away or excess to avoid some operating expenses. Watervliet projected that it could save over $250,000 within the first year as a result of these actions. Officials at Watervliet stated that they were moving forward with these actions. Additionally, in December 2013, AMC conducted an analysis of personnel levels at Rock Island Arsenal and recommended that Rock Island reduce its workforce from its approximately 1,200 personnel to between 500 and 600 personnel. According to AMC officials, this reduction has not been made. However, unless DOD first identifies the manufacturing arsenals’ critical capabilities, it will be unable to determine whether such adjustments of equipment and personnel levels will enable the arsenals to sustain those capabilities.

DOD’s Report Addresses Statutory Reporting Elements, but Additional Information Would Have Made It More Consistent with Relevant Research Presentation Standards

DOD’s September 2014 Report on Manufacturing Arsenal Study met the statutory requirements to address seven different reporting elements. However, we found that additional information and coordination would have made the report more consistent with relevant generally accepted research presentation standards for a defense research study, and therefore would have helped decision makers to identify and evaluate the information presented.

DOD submitted its report to Congress in response to the mandate in section 322 of the National Defense Authorization Act for Fiscal Year 2014, which required the Secretary of Defense to conduct a one-time review of the manufacturing arsenals, covering the seven elements specified in the statute. The report was to include the results of reviews of 1. current and expected manufacturing requirements across the military services and Defense Agencies, to identify critical manufacturing competencies and supplies, components, end items, parts, assemblies, and sub-assemblies for which there is no or limited domestic commercial source and which are appropriate for manufacturing within an arsenal owned by the United States in order to support critical manufacturing capabilities; 2. how DOD can more effectively use and manage public-private partnerships to preserve critical industrial capabilities at the manufacturing arsenals for future national security requirements, while providing the Department of the Army with a return on its investment; 3. the effectiveness of the strategy of DOD to assign work to be performed at each of the arsenals and the potential for alternative strategies that could better identify work to be performed at each arsenal; 4. the impact of the rate structure driven by the Department of the Army’s working capital funds on public-private partnerships at each arsenal; 5. the extent to which operations at each arsenal can be streamlined, improved, or enhanced; 6. the effectiveness of the implementation by the Department of the Army of cooperative agreements, authorized at manufacturing arsenals under section 4544 of title 10, United States Code; and 7. mechanisms within DOD for ensuring that appropriate consideration is given to the unique manufacturing capabilities of arsenals for manufacturing requirements of DOD for which there is no or limited domestic commercial capability.

Based on our review of the report, we determined that DOD met the statutory requirements, because the report includes each of the seven elements and related content. The 5-page report contains subsections specific to each element, primarily within the context of the cost recovery requirements of the Working Capital Fund and the declining workload experienced by the arsenals. For example, in response to the second element regarding how DOD can more effectively use and manage public-private partnerships, DOD reported that, while it will continue to rely on existing legal authorities for public-private partnerships, more effective use of such partnerships requires the stabilization of arsenal rates and the use of alternate rate structures. The report states the arsenals have limited flexibility to adjust their rates and that the need for the arsenals to charge a fully burdened rate that recovers past operating losses causes them to lose potential opportunities for public-private partnerships. DOD also stated that it can more effectively utilize public- private partnerships by increasing the military departments’ and defense agencies’ knowledge of the arsenals’ capabilities through various forums, such as industry days and DOD maintenance symposiums. Additionally, for the fourth element, regarding the impact of the rate structure driven by the Department of the Army’s Working Capital Funds on the public-private partnerships at each arsenal, DOD reported that the arsenals must recover their full costs from customer charges, since they are working capital-funded activities. DOD explained that some of the costs borne by the arsenals are attributed to capacity required for surge operations and some are for workforce in excess of workload. The report states that, when the full costs are apportioned over a small workload base, the impact is higher rates, which discourages partners and customers from using the arsenals. DOD’s report reiterated that the need for arsenals to charge a fully burdened rate that recovers past operating losses causes them to lose potential opportunities for public-private partnerships.

While DOD’s report met the statutory requirements, we determined that DOD could have taken actions that we believe would have made the report more consistent with relevant generally accepted research presentation standards for a defense research study and, therefore, would have made the presented results more useful to decision makers. Specifically, we found that DOD’s presentation of most of the reporting elements could have been more sound, complete, and clear, which would have facilitated decision makers’ evaluation of the information presented. Furthermore, DOD could have coordinated the results of its study with participants and stakeholders—and obtained and considered their comments—before finalizing its report, to better ensure its soundness, completeness, and clarity.

Generally accepted research standards for a defense research study define a sound and complete defense research study as one that provides, among other things, timely, complete, and relevant information for the client and stakeholders. Of these, there is a subset of standards for presenting the results from such a study. The extent to which a report’s presentation of results is consistent with these relevant standards is an indication of the ease with which the evidence can be evaluated and of the soundness and completeness of the report and, thus, its usefulness in enabling decision makers to make fully informed decisions. These GAO-developed standards are consistent with current Office of Management and Budget and DOD guidelines on ensuring and maximizing the quality of information disseminated to the public. We determined that the following presentation standards are relevant, given our objectives and the content of DOD’s report: 1. Does the report present an assessment that is well documented and conclusions that are supported by the analyses? 2. Are the report’s conclusions sound and complete? 3. Are the study results presented in a clear manner? 4. Are study participants/stakeholders informed of the study results and recommendations?

In applying the first three standards on the presentation of results, which describe the soundness, completeness, and clarity of the information presented, we found that DOD’s report was consistent with those standards for two of the seven reporting elements. First, the report details how rate structures that are driven by the Army Working Capital Fund impact DOD’s ability to maintain public-private partnerships at the manufacturing arsenals. In doing so, it clearly describes how the manufacturing arsenals must charge a rate to their public partners and customers that is determined by the Army Working Capital Fund, and how that rate can increase when work to be performed at the arsenals decreases. According to DOD’s report, higher rates can lead to the loss of current and potential opportunities for public-private partnerships at the manufacturing arsenals. Second, the report discusses the extent to which operations at each manufacturing arsenal could be streamlined, improved, or enhanced. In doing so, the report describes in detail the limited flexibility the manufacturing arsenals have to streamline, improve, or enhance their operations. For example, the report notes that, given that the manufacturing arsenals are restricted in their ability to conduct reductions in force to adjust their personnel levels, they are limited to using hiring freezes and voluntary early retirements or separations to decrease personnel levels in times of decreased workloads.

For the remaining five reporting elements, we found that DOD’s September 2014 report is not consistent with the relevant presentation standards for soundness, completeness, and clarity. For example, to address the seventh reporting element—that DOD identify mechanisms for ensuring that the manufacturing arsenals are considered as a source of manufacture—DOD’s report notes that the Secretary of the Army directed AMC to work with DLA to make the Army manufacturing arsenals a source of manufacture for the Army parts within the DLA inventory. DOD’s report, however, does not disclose to what extent this has resulted in the manufacturing arsenals actually being considered as a source and does not discuss any challenges in doing so, such as issues associated with guidance for conducting make-or-buy analyses, discussed earlier in this report. Furthermore, to address the first reporting element—to review the manufacturing arsenals’ current and expected requirements to support their critical capabilities—DOD’s report identified a list of current and expected manufacturing requirements for items that it designated as appropriate for production in one of the manufacturing arsenals, but the report does not provide a clear explanation of how DOD identified these items and does not identify which military services or defense agencies these requirements apply to. Additionally, the report does not disclose that, as previously discussed, DOD has not developed and implemented a process for identifying the manufacturing arsenals’ critical capabilities. Moreover, to address the second reporting element—that DOD discuss how it can more effectively use and manage public-private partnerships— DOD’s report explains how the department is currently using these partnerships but does not clearly explain how any suggested improvements would provide an additional return on investment to the Army or how the use of public-private partnerships would aid in preserving the manufacturing arsenals’ critical capabilities.

Table 1 summarizes our assessment of what DOD included in its report that is consistent with relevant defense research presentation standards. For the five reporting elements for which we found that DOD was not consistent with these standards, the table provides examples of additional information that we determined could have been included to make the report more consistent with the relevant presentation standards for soundness, completeness, and clarity.

In addition, we determined that DOD’s report was not consistent with the generally accepted research standard that participants or stakeholders be informed of the defense study’s results and recommendations. Officials from a DOD office mentioned in the September 2014 report told us that no one from their office either participated in the study or reviewed the report prior to its publication. They explained, after we provided them a copy of the report, that one of the analyses that was reportedly conducted by their office—and mentioned in DOD’s report—was not conducted in the manner described or for the purposes indicated in the report. They further explained that had they reviewed the information in the report about the analysis their office conducts, such a misstatement would not have occurred. Additionally, officials from relevant DOD components— including the three manufacturing arsenals and their higher headquarters—told us that they had not been given the opportunity to review or comment on the final version of the report before it was issued. For example, when we spoke with officials at the manufacturing arsenals and at some of the headquarters organizations, they told us that they had not seen the issued report until we showed it to them. Moreover, AMC officials told us that the manufacturing arsenals had not been given an opportunity to review the information used to support the report to confirm that the details in the final report were complete.

When we discussed the results of our assessment of the September 2014 report with ODASD (MPP) officials who had the lead for developing the report, they disagreed with our assessment related to relevant generally accepted research presentation standards. In our June 2015 meeting, they explained that they had addressed the statutory requirements and that was sufficient, questioning the need to follow the generally accepted defense research presentation standards we determined to be relevant in assessing DOD’s report. We agree that DOD’s report met the statutory requirements by including a discussion of each of the seven reporting elements. However, we also believe that it is appropriate to apply the relevant generally accepted defense research standards for the presentation of results, because consistency with these standards helps to indicate the extent to which the results presented in the report are useful to decision makers. Moreover, as previously mentioned, the relevant generally accepted research presentation standards we used to assess DOD’s report are consistent with Office of Management and Budget guidelines and DOD guidance. ODASD (MPP) officials did not provide any examples where we had overlooked information in the report that our assessment determined could have been included to make it more sound, complete, and clear for use by decision makers. Further, these officials did not disagree with our assessment that the report had not been shared or coordinated with participants and stakeholders.

Conclusions

As a result of the decline in demand for materiel, DOD is facing challenges in assigning work to its three manufacturing arsenals. DOD has taken various actions in an effort to assign work to the manufacturing arsenals, but these actions collectively have not resulted in the arsenals generating sufficient revenue to recover their operating expenses. The effectiveness of these actions has been limited in part by the fact that DOD has not developed clear, step-by-step implementing guidance on how to conduct make-or-buy analyses, which would help to ensure that the arsenals are appropriately considered as a source of manufacture. Because the arsenals are generating insufficient revenue, Congress has provided $375 million collectively in the prior and current fiscal years to help recover operating losses and maintain competitive rates. Unless the manufacturing arsenals are able to generate sufficient revenue to recover their operating expenses, it is likely that they will need continued funding or will need to make adjustments to personnel and equipment levels to reduce their operating expenses and maintain competitive rates.

DOD is not strategically positioned to sustain the manufacturing arsenals’ critical capabilities. These critical capabilities help ensure that DOD is able to respond to national emergencies and obtain products and services that it could not otherwise acquire from private industry in an economical manner. While there is a strategic plan that covers the manufacturing arsenals and has established related goals and objectives, DOD has not identified or documented fundamental elements, such as time frames and resources, for implementing the plan. In not identifying and documenting these fundamental elements, DOD is inconsistent in applying standard practices for project management and, therefore, lacks information that would be useful in determining whether progress is being made in achieving the plan’s goals and objectives. More importantly, DOD cannot achieve the strategic plan’s goals and objectives until it has identified the manufacturing arsenals’ critical capabilities. After falling short in prior efforts, DOD has an effort under way to develop a process to identify critical capabilities and a method for determining the minimum workload needed to sustain them, but that effort has been delayed. As a result, it is not clear when DOD will be able to act on its intention to develop and issue guidance—such as a DOD instruction—to implement the process and method being developed. Until such an instruction is issued, DOD will continue to lack a process for identifying the manufacturing arsenals’ critical capabilities and will not be positioned to determine the minimum amount of work or the levels of equipment and personnel needed to sustain the arsenals’ capabilities.

With the issuance of its September 2014 report on the manufacturing arsenals, DOD met statutory requirements. However, we determined that additional information would have made the report more consistent with relevant generally accepted research presentation standards for a defense study. Additionally, had DOD coordinated its results with participants and stakeholders, they could have provided comments or corrections to misstatements, as needed. Doing so would have enabled DOD to present a more sound, complete, and clear report that not only would have met statutory requirements, but would have been more useful to Congress in its oversight of DOD’s manufacturing arsenals. Because DOD’s report was prepared in response to a one-time, nonrecurring mandate, we are not making any recommendations to amend the report or provide additional detail.

Recommendations for Executive Action

To help DOD ensure that it appropriately considers the manufacturing arsenals as a source of manufacture and is strategically positioned to sustain the manufacturing arsenals’ critical capabilities, we recommend that the Secretary of Defense direct

The Secretary of the Army to issue clear, step-by-step implementing guidance, such as an instruction or guidebook, on the process for conducting make-or- buy analyses in a consistent manner and identify and document fundamental elements—such as steps, interim milestones, time frames, and resources—for implementing the Army’s Organic Industrial Base Strategic Plan 2012-2022 and

The Office of the Deputy Assistant Secretary of Defense for Maintenance Policy and Programs—in coordination with the military services, as appropriate, to complete DOD’s ongoing effort to establish a process for identifying the manufacturing arsenals’ critical capabilities and a method for determining the minimum workload needed to sustain these capabilities and develop and issue guidance, such as a DOD instruction, to implement the process for identifying the manufacturing arsenals’ critical capabilities and the method for determining the minimum workload needed to sustain these capabilities.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. In written comments, DOD concurred with all of our recommendations. DOD’s comments are summarized below and reprinted in appendix II. DOD also provided technical comments which we have incorporated into our report as appropriate.

In addition to its overall concurrence with our recommendations, DOD stated that it does not agree with the implication that these steps will lead to the provision of sufficient revenue to cover all of the manufacturing arsenals’ expenses. However, we did not state or imply that implementation of these recommendations will increase revenue. Rather, the recommendations’ stated intent is to ensure that DOD appropriately considers the manufacturing arsenals as a source of manufacture and is strategically positioned to sustain their critical capabilities. As explained in our report, until DOD determines the manufacturing arsenals’ critical capabilities, it will not be positioned to determine the minimum amount of work or the levels of equipment and personnel needed to sustain those capabilities.

DOD concurred with our recommendation related to issuing implementing guidance on make-or-buy analyses but provided no details on how or when it would issue such guidance. Further, DOD explained that it did not agree with the implication that make-or-buy analyses would necessarily increase revenue provided to arsenals and noted that the process may result in reduced revenue. We did not state or imply that the issuance of implementing guidance on the process for conducting make-or-buy analyses would increase revenue. Rather, we believe that in the absence of clear, step-by-step implementing guidance on how to conduct make-or- buy analyses, DOD cannot provide reasonable assurance that it is appropriately considering the manufacturing arsenals as a potential source of manufacture. As stated in our report, although implementing such guidance would not guarantee that the manufacturing arsenals receive sufficient workload, it would ensure that they are appropriately considered.

DOD concurred with our recommendation related to implementing its 2012 strategic plan but provided no details on how or when it would implement the recommendation. DOD also stated that, given the overall constraints placed on the department’s budget under the Budget Control Act of 2011, as amended, it cannot guarantee the availability of any resources that it would identify by implementing this recommendation. Our report discusses how standard practices for project management call for agencies to conceptualize, define, and document specific goals and objectives in their planning processes and to identify the appropriate steps, milestones, time frames, and resources they need to achieve those goals and objectives. However, DOD’s strategic plan does not contain any of these fundamental elements. Unless DOD identifies and documents fundamental elements by including information that would be useful in determining its progress toward achieving its stated goals and objectives, the department will not be strategically positioned to sustain the manufacturing arsenals’ critical capabilities in any budget environment.

DOD concurred with our recommendation related to developing a process to identify the arsenals’ critical capabilities and a method to determine the minimum workload needed to sustain those capabilities. DOD stated that its effort to address this recommendation is ongoing, but added it does not agree that developing such a process will result in sufficient revenue to cover arsenal expenses. We did not state or imply that establishing a process for identifying the manufacturing arsenals’ critical capabilities and a method for determining the minimum workload needed to sustain these capabilities would result in sufficient revenue to cover arsenal expenses. Rather, we believe such a process is needed to help ensure that DOD is strategically positioned to sustain the manufacturing arsenals’ critical capabilities and achieve its 2012 strategic plan’s goals and objectives. Although DOD did not specifically comment on our recommendation related to issuing guidance to implement a process for identifying the arsenals’ critical capabilities and a method for determining the minimum workload needed to sustain these capabilities, it commented that it expects to issue an instruction incorporating such a process by the end of fiscal year 2016. We believe that, if fully implemented, these actions should address our recommendations and strategically position the department to sustain the manufacturing arsenals’ critical capabilities.

In its comments, DOD also took issue with how we characterized Congress “‘providing’ additional funds” to cover the excess of arsenal expenses over revenue. DOD stated that no additional funds were added to the DOD budget for this shortfall; rather, Congress “redirected funding” from other essential defense missions. We did not state or imply that amounts appropriated for arsenal expenses were additional to DOD’s overall budget authority for fiscal years 2014 and 2015. Our discussion of funding for Industrial Mobilization Capacity and the Arsenal Sustainment Initiative highlighted specific appropriations made by Congress to the Working Capital Fund for the express purposes of covering the costs of unutilized or underutilized plant capacity through the Working Capital Fund’s Industrial Mobilization Capacity subaccount in fiscal year 2014 and maintaining competitive rates at arsenals through the Arsenal Sustainment Initiative in fiscal year 2015. We did not discuss these appropriations in the context of DOD’s overall budget authority, or state or suggest that they were derived from a concurrent increase in budget authority.

Finally, DOD did not agree with our assessment of DOD’s September 2014 report on the manufacturing arsenals against four generally accepted research presentation standards, questioning why the presentation standards are more relevant for this particular DOD report to Congress than they are for the many reports submitted each year. We do not believe that the presentation standards we applied are more relevant for the September 2014 report than for any other report DOD submits to Congress. We believe that all reports submitted by federal agencies to Congress should not only comply with applicable statutory reporting requirements, as was the case with the September 2014 report but should also follow applicable generally accepted research standards. As discussed in our methodology, we determined that the following four standards, which focused on the presentation of results, were applicable as DOD did not have a documented method for its study and given the contents of the September 2014 report: the report presents an assessment that is well documented and conclusions that are supported by the analyses; the report’s conclusions are sound and complete; the study results are presented in a clear manner; and study participants/stakeholders are informed of the study results and recommendations.

In its comments, DOD did not provide any explanation as to why it believes these standards should not have been applied in our assessment of the September 2014 report.

DOD further noted in its comments that the application of the generally accepted research presentation standards would not have altered the September 2014 report or its contents. We disagree. As discussed in the report, we identified multiple examples of additional information that we determined could have been included to make the report more consistent with the relevant presentation standards for soundness, completeness, and clarity. DOD also stated that we are not correct in our assertion that the report was not shared or coordinated with participants and stakeholders, explaining that the September 2014 report was fully staffed with the Army’s Assistant Secretary for Acquisition, Logistics and Technology; the Deputy Chief of Staff for Logistics; and the Army Materiel Command. We do not disagree that the report was coordinated with the organizations identified in DOD’s comments. However, we identified other key stakeholders with whom the report was not coordinated. Most notably, we found that the manufacturing arsenals had not been given an opportunity to review the information used to support the report or to review the final version of the report. Had DOD adhered to relevant generally accepted research presentation standards for a defense study in preparing the September 2014 report, it could have better ensured the report’s soundness, completeness, and clarity and therefore, its usefulness to decision makers.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretaries of the Army, Air Force, and Navy. The report also is available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-5741 or ayersj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report assesses (1) actions, if any, that the Department of Defense (DOD) has taken to assign work to the manufacturing arsenals to generate sufficient revenue to recover their operating expenses; (2) the extent to which DOD is strategically positioned to sustain the manufacturing arsenals’ critical capabilities; and (3) the extent to which DOD’s September 2014 report meets the requirements to address the statutory reporting elements and is consistent with relevant generally accepted research presentation standards for a defense research study. To address these reporting objectives, we visited or contacted knowledgeable officials with responsibilities related to arsenal operations from the following organizations: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics

Office of the Deputy Assistant Secretary of Defense for Manufacturing

Defense Procurement and Acquisition Policy

Office of the Assistant Secretary of Defense for Logistics and Materiel Readiness

Office of the Principal Deputy Assistant Secretary of Defense for Maintenance Policy and Programs

Office of the Deputy Assistant Secretary of Defense for Defense Finance and Accounting Service Defense Manpower Data Center Deputy Assistant Secretary of the Army for Acquisition Policy and Logistics

Office of the Assistant Secretary of the Army for Acquisition, Logistics

Department of the Army, Headquarters / G4, G8

Army Materiel Command / G1, G3, G4, G8 Joint Munitions Command Life Cycle Management Command

Pine Bluff Arsenal

TACOM Life Cycle Management Command

Rock Island Arsenal Joint Manufacturing and Technology

Watervliet Arsenal Joint Manufacturing and Technology

Army Program Executive Office Ground Combat Systems

Abrams Tank Program Office We also obtained pertinent documents, including DOD directives, instructions, and reports; Army regulations and instructions, memorandums, strategic plans, and other guidance; and information on the organic defense industrial base and each of the three manufacturing arsenals (Pine Bluff, Rock Island, and Watervliet), such as the arsenals’ critical capabilities and current levels of workload. Additionally, we discussed personnel and workload data with subject matter experts at the Defense Manpower Data Center and the Defense Finance and Accounting Service. To provide background information on recent trends in workload at the three manufacturing arsenals, we summarized direct labor hour data compiled by Army Materiel Command officials for civilian and contract personnel from fiscal years 2000 through 2014. We assessed the reliability of the direct labor hour data we obtained from Army Materiel Command through interviews with knowledgeable officials and determined that these data were sufficiently reliable to use in this report for this limited purpose.

For our first objective, to assess the actions, if any, that DOD has taken to assign work to the manufacturing arsenals to generate sufficient revenue to recover their operating expenses, we also interviewed DOD officials who were involved in assessing and implementing efforts to improve or enhance operations at the arsenals. We compared existing guidance on the process used to consider manufacturing arsenals as a source of manufacture to federal internal control standards for control activities contained in GAO’s Standards for Internal Control in the Federal Government.

For our second objective, to assess the extent to which DOD is strategically positioned to sustain the manufacturing arsenals’ critical capabilities, we interviewed DOD officials who contributed significantly to the department’s current strategy to assign work to be performed at the arsenals. We then compared DOD’s existing strategy for the manufacturing arsenals to standard practices for project management and identified discrepancies. We also reviewed two Army assessments related to the levels of equipment and personnel and determined that one of the assessments described an approach and findings that were reasonable, but we did not assess the accuracy or reliability of the underlying data because doing so was beyond the scope of this review. The other assessment, however, did not contain sufficient information for us to determine if the approach used to calculate its results was reasonable. In the absence of other reliable sources, we limited the use of these three assessments in the report to noting that the arsenals had conducted assessments containing recommendations intended to guide subsequent decision making.

For our third objective, to determine the extent to which DOD’s September 2014 report meets the requirements to address the statutory reporting elements and is consistent with relevant generally accepted research presentation standards for a defense research study, we conducted a two-part assessment of DOD’s September 2014 report. First, to assess the extent to which DOD’s September 2014 report meets the statutory requirement to address the seven reporting elements, we compared the report to the elements listed in section 322 of the National Defense Authorization Act for Fiscal Year 2014. For each reporting element, we determined whether DOD’s report met the statutory requirement by including the element and providing related content. Second, we assessed the extent to which DOD’s September 2014 report is consistent with relevant generally accepted research presentation standards for a defense research study. To do so, we determined which generally accepted research presentation standards for a sound, complete, and clear defense research study were relevant to the contents of the report, given our objective. In 2006, we described these standards in a report on DOD transportation capabilities. In this 2006 report, we reviewed research literature and DOD guidance and identified frequently occurring, generally accepted research standards that are relevant for defense studies, including those related to the presentation of results. These GAO-developed generally accepted research presentation standards are consistent with Office of Management and Budget guidelines and DOD guidance on ensuring and maximizing the quality of information disseminated by federal agencies to the public. We identified 36 generally accepted research standards for a defense research study in the areas of design, execution, and presentation of results. We determined that these standards are still current and relevant for the purposes of this report. Because DOD did not have a documented method for its study, we did not assess DOD’s September 2014 report against the generally accepted research standards for design (14 standards) and execution of the design (15 standards). Consequently, we confined our objective and assessment to the subset of 7 standards for presenting the results of a defense study. Of these 7 standards related to the presentation of results, we determined, based on the content of DOD’s report, that the following 4 were relevant: 1. Does the report present an assessment that is well documented and conclusions that are supported by the analyses? 2. Are the report’s conclusions sound and complete? 3. Are the study results presented in a clear manner? 4. Are study participants/stakeholders informed of the study results and recommendations?

We determined that the remaining three research presentation standards were not relevant, given our review’s objectives and based on the content of DOD’s report. For example, since we separately assessed the extent to which DOD’s report met the statutory requirements to address the statutory reporting elements, we did not assess whether DOD’s report addressed its objectives in the context of the generally accepted defense research presentation standards, as this would have been duplicative. Also, because the report did not include recommendations or provide options, we did not apply the generally accepted defense research presentation standards on whether recommendations were supported by analyses or whether a realistic range of options was provided.

After determining the relevant presentation standards, we then compared the contents of DOD’s September 2014 report and available supporting documentation—such as DOD policy, guidance, assessments, and briefings—to the 4 relevant research presentation standards. The extent to which the report’s presentation of results is consistent with these relevant standards is an indication of the ease with which the evidence can be evaluated and of the soundness and completeness of the report and, thus, its usefulness in enabling decision makers to make fully informed decisions. We considered DOD’s response to a statutory reporting element to be consistent with relevant generally accepted defense research presentation standards when the report explicitly addressed (e.g., included meaningful facts, figures, or clearly discussed) all aspects of the element and included sufficient specificity and detailed support. We considered DOD’s response to a reporting element to be inconsistent with these standards when the report neither explicitly addressed all aspects of the element nor included sufficient specificity and detailed support. In such cases, we provided examples of additional information that, although not statutorily required, we believe would have made the report more consistent with the 4 relevant generally accepted research presentation standards. In addition, we discussed the results of our assessment of the September 2014 report with ODASD (MPP) officials—who had the lead for developing the report—and obtained their perspectives regarding the approach they used to develop it, including their rationale for (1) choosing the level of detail they provided for particular reporting elements in the report and (2) not sharing the report with participants and stakeholders before it was issued.

We conducted this performance audit from June 2014 to November 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff members made key contributions to this report: Larry J. Junek, Assistant Director; Yecenia C. Camarillo, Timothy J. Carr, Patricia Farrell Donahue, Cynthia L. Grant, Joanne Landesman, Amie Lesser, Felicia M. Lopez, Oscar W. Mardis, Sabrina C. Streagle, and Matthew R. Young.

Related GAO Products

Regional Missile Defense: DOD’s 2014 Report Generally Addressed Required Reporting Elements, but Excluded Additional Key Details. GAO-15-32. Washington, D.C.: December 1, 2014.

Army Industrial Operations: Budgeting and Management of Carryover Could Be Improved. GAO-13-499. Washington, D.C.: June 27, 2013.

Defense Logistics: Oversight and a Coordinated Strategy Needed to Implement the Army Workload and Performance System. GAO-11-566R. Washington, D.C.: July 14, 2011.

Defense Infrastructure: The Army Needs to Establish Priorities, Goals, and Performance Measures for Its Arsenal Support Program Initiative. GAO-10-167R. Washington, D.C.: November 5, 2009.

Depot Maintenance: Improved Strategic Planning Needed to Ensure That Army and Marine Corps Depots Can Meet Future Maintenance Requirements. GAO-09-865. Washington, D.C.: September 17, 2009.

Depot Maintenance: DOD’s Report to Congress on Its Public-Private Partnerships at Its Centers of Industrial and Technical Excellence (CITEs) Is Not Complete and Additional Information Would Be Useful. GAO-08-902R. Washington, D.C.: July 1, 2008.

Defense Transportation: Study Limitations Raise Questions about the Adequacy and Completeness of the Mobility Capabilities Study and Report. GAO-06-938. Washington, D.C.: September 20, 2006 Depot Maintenance: Public-Private Partnerships Have Increased, but Long-Term Growth and Results Are Uncertain. GAO-03-423. Washington, D.C.: April 10, 2003.

Army Industrial Facilities: Workforce Requirements and Related Issues Affecting Depots and Arsenals. GAO/NSIAD-99-31. Washington, D.C.: November 30, 1998.