Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD's Acquisition Outcomes
Published: 2007-03-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Over the next several years, the Department of Defense (DOD) plans to invest $1.4 trillion in major weapons programs. While DOD produces superior weapons, GAO has found that the department has failed to deliver weapon systems on time, within budget, and with desired capabilities. While recent changes to DOD's acquisition policy held the potential to improve outcomes, programs continue to experience significant cost and schedule overruns. GAO was asked to examine how DOD's processes for determining needs and allocating resources can better support weapon system program stability. Specifically, GAO compared DOD's processes for investing in weapon systems to the best practices that successful commercial companies use to achieve a balanced mix of new products, and identified areas where DOD can do better. In conducting its work, GAO identified the best practices of: Caterpillar, Eli Lilly, IBM, Motorola, and Procter and Gamble.
section_title: What GAO Found
paragraphs: To achieve a balanced mix of executable development programs and ensure a good return on their investments, the successful commercial companies GAO reviewed take an integrated, portfolio management approach to product development. Through this approach, companies assess product investments collectively from an enterprise level, rather than as independent and unrelated initiatives. They weigh the relative costs, benefits, and risks of proposed products using established criteria and methods, and select those products that can exploit promising market opportunities within resource constraints and move the company toward meeting its strategic goals and objectives. Investment decisions are frequently revisited, and if a product falls short of expectations, companies make tough go/no-go decisions. The companies GAO reviewed have found that effective portfolio management requires a governance structure with committed leadership, clearly aligned roles and responsibilities, portfolio managers who are empowered to make investment decisions, and accountability at all levels of the organization. In contrast, DOD approves proposed programs with much less consideration of its overall portfolio and commits to them earlier and with less knowledge of cost and feasibility. Although the military services fight together on the battlefield as a joint force, they identify needs and allocate resources separately, using fragmented decision-making processes that do not allow for an integrated, portfolio management approach like that used by successful commercial companies. Consequently, DOD has less assurance that its investment decisions address the right mix of warfighting needs, and, as seen in the figure below, it starts more programs than current and likely future resources can support, a practice that has created a fiscal bow wave. If this trend goes unchecked, Congress will be faced with a difficult choice: pull dollars from other high-priority federal programs to fund DOD's acquisitions or accept gaps in warfighting capabilities.

section_title: Letter
paragraphs: Although the Department of Defense (DOD) produces the best weapons in  the world, it has not been able to deliver planned systems on time and  within budget. It is not unusual to see cost increases that add up to tens or  hundreds of millions of dollars, schedule delays that add up to years, and  large and expensive programs being scrapped after years of failing to  achieve promised capabilities. While recent changes to DOD’s acquisition  policy held the potential to improve such outcomes, programs have  continued to experience significant cost and schedule overruns and  performance shortfalls. Over the next several years, DOD plans to invest  $1.4 trillion in major weapon system programs—doubling what it planned  to spend on such programs 5 years ago. Continued failure to deliver  weapon systems on time and within budget not only delays providing  critical capabilities to the warfighter, but results in less funding being  available for other DOD and federal priorities.
In the commercial market, effectively developing and marketing new  products is fundamental to the continued growth and success of  companies. Without a steady stream of product innovations to meet  evolving market needs, companies are likely to see their sales and profits  fall. At the same time, if the products in development outstrip their  resources or do not meet customer needs, companies can face financial  ruin. Several recent studies issued by leading thinkers in the area of  product innovation and development have reported that leading  commercial companies achieve success in product development by  following a disciplined process for ensuring they have the right mix of new  products that meet customer needs within available resources.
In fiscal year 2006, the Senate Armed Services Committee raised concerns  that DOD’s poor track record with acquisition programs was linked not  only to the department’s Defense Acquisition System (DAS) for managing  product development, but also to the department’s Joint Capabilities  Integration and Development System (JCIDS) for identifying the  warfighters’ needs and the Planning, Programming, Budgeting and  Execution (PPBE) process for allocating resources. Consequently, the  Committee directed GAO to examine how DOD’s needs identification and  resource allocation processes can better support program stability in  major weapon systems acquisition. This report (1) identifies best  practices of successful commercial companies for ensuring that they  pursue the right mix of programs to meet the needs of their customers  within resource constraints and (2) compares DOD’s enterprise-level  processes for investing in weapon systems to these practices.
To identify best practices of successful companies, we reviewed related  professional and academic publications, and interviewed knowledgeable  officials from five successful commercial companies: Caterpillar, Eli Lilly,  IBM, Motorola, and Procter & Gamble. While the products developed by  these companies range from heavy construction equipment and high-end  electronics to pharmaceuticals and household items, each of the  companies manages a large diversified portfolio of products, spends  billions of dollars annually on research and development, and has  thousands of employees worldwide. To examine DOD’s processes for  making investment decisions, we reviewed related legislation and DOD  directives, instructions, and guidance; conducted interviews with and  received briefings from relevant Joint Staff, Office of the Secretary of  Defense (Comptroller, Program Analysis and Evaluation, and Acquisition,  Technology and Logistics), and other government officials; reviewed  current literature assessing DOD’s decision-making processes; and  analyzed DOD requirements documents. We compared DOD’s enterprise- level practices to commercial best practices to identify potential areas for  improvement. For additional details on how we performed our review, see  appendix I. Our work was conducted between March 2006 and February  2007 in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: DOD’s programs for acquiring major weapon systems have taken longer,  cost more, and delivered fewer quantities and capabilities than planned.  We have documented these problems for decades. Most recently, we  reported that 27 major weapon programs we have assessed since they  began product development have experienced cost increases of nearly   34 percent over their original research, development, test, and evaluation  (RDT&E) estimates, and increases of almost 24 percent in acquisition  cycle time (see table 1).
When cost and schedule problems occur in one program, DOD often  attempts to pay for the poorly performing program by taking funds from  others. Doing so has destabilized other programs and reduced the overall  buying power of the defense dollar as DOD and the military services are  forced to cut back on planned quantities or capabilities to stay within  budget limitations. The F-22A Raptor program is a case in point: As costs  escalated in the program, the number of aircraft the Air Force planned to  buy was drastically reduced from 648 to 183. Similarly, as the Joint  Tactical Radio System (JTRS) encountered development problems, the  number of requirements was reduced or deferred by about one-third. As a  result, several programs that were dependent on JTRS also had to make  adjustments and go forward with alternative, less capable solutions.  DOD’s approach to managing weapon system investments ultimately  results in less funding being available for other competing needs in DOD  as well as other federal priorities, as the expenditure of tax dollars within  DOD reduces the amount of funding available for those priorities.
Taking into account the differences between commercial product  development and weapons acquisitions, we have recommended that DOD  adopt a knowledge-based, incremental approach to developing and  producing weapon systems. This type of an approach requires program  officials to demonstrate that critical technologies are mature, product  designs are stable, and production processes are in control at key  junctures in the acquisition process.
DOD has three major processes involved in making weapon system  investment decisions. These processes, depicted in figure 1, are the Joint  Capabilities Integration and Development System (JCIDS), for identifying  warfighting needs; the Planning, Programming, Budgeting and Execution  (PPBE) system, for allocating resources; and the Defense Acquisition  System (DAS), for managing product development and procurement.  Much of our prior work has focused on identifying commercial best  practices that could be used to improve the Defense Acquisition System— from the point just before product development starts onward. In this  report, however, we look at earlier stages in DOD’s investment process— from the point where gaps in warfighting capability are assessed in JCIDS  through the point where alternative solutions to resolve those gaps are  analyzed under the DAS (see fig. 1).
section_title: Successful Companies Take a Disciplined, Integrated Approach to Prioritize Market Needs and Initiate a Balanced Mix of Executable Development Programs
paragraphs: To ensure they achieve a balanced mix of executable development  programs, the successful commercial companies we reviewed use a  disciplined and integrated approach to prioritize market needs and  allocate resources. This approach, known as portfolio management,  requires companies to view each of their investments from an enterprise  level as contributing to the collective whole, rather than as independent  and unrelated. With this enterprise viewpoint, companies can effectively  (1) identify and prioritize market opportunities and (2) apply available  resources to potential products to select the best mix of products to  exploit the highest priority—or most promising—opportunities.  Ultimately, each of the companies we reviewed seeks to achieve a  balanced portfolio that maximizes the return on investments and moves  the company toward achieving its strategic goals and objectives. This type  of approach depends on strong governance with committed leadership,  clearly aligned responsibility, and effective accountability at all levels of  the organization.
As depicted in figure 2, a portfolio management approach begins with an  enterprise-level identification and definition of market opportunities and  then the prioritization of those opportunities within resource constraints.  Once opportunities have been prioritized, companies draft initial business  cases for alternative product ideas that could be developed to exploit each  of the highest priority opportunities. Each alternative product proposal— represented by a black dot—enters a gated review process. At each review  gate, product proposals are assessed against corporate resources,  established criteria, competing products, and the goals and objectives of  the company as a whole. As alternatives pass through each review gate,  the number is expected to decrease, until only those alternatives with the  greatest potential to succeed make it into the product portfolio.
subsections: section_title: Identifying and Prioritizing Market Opportunities Lays the Foundation for Achieving the Right Mix of Products
paragraphs: To make informed decisions about what market opportunities to target,  the companies we reviewed first establish a strategy that lays out the  overall goals, objectives, and direction for the company. As part of their  strategy, companies identify enterprise-level sales and profit targets,  strategic business areas they want to focus on, the extent to which current  products and new development efforts will support their growth  objectives, and how they will allocate resources across business units and  functional areas. This strategy provides a framework for the companies’  investment decisions. Within this framework, companies conduct a series  of market analyses to develop a comprehensive understanding of the  market environment, including product trends, technology trends, and  customer needs.
IBM for example, follows a structured market planning process to identify,  prioritize, and target attractive market segments. The first phase of this  process, called Market Definition, focuses on understanding the  marketplace, including identifying potential customers and their needs.  During this phase, IBM examines the marketplace and technology  environments and identifies attractive market segments that contain  potential market opportunities—where customer wants or needs exist.  Each segment is categorized into one of four areas based on needs of the  customers and the company’s product offerings (see fig. 3): “strike zone,”  “traditional,” “pushing the envelope,” and “white space.” The strike zone  represents IBM’s core business—market segments where IBM has an  established customer base that it is successfully serving with existing  product offerings. In contrast, white space represents market segments of  new customers with wants and needs that are new and different for IBM.  White space opportunities often require discovery and innovation. The  traditional and pushing the envelope areas fall between the strike zone and  white space. Traditional opportunities exist when new customers could be  attracted to an existing market—one IBM is already active in—by  modifying or enhancing existing products or services. Pushing the  envelope opportunities exist where the needs of current customer groups  move them into a new market segment. These attractive market segments  are prioritized during the next phase of IBM’s process, known as the  Capability Assessment phase. During this phase each segment’s overall  attractiveness and potential profitability are assessed, along with IBM’s  available resources—like capital, cash, and current products—and its  competitive position within each segment. This analysis leads to the  selection of targeted market segments.
Motorola emphasizes the importance of targeting the right market  segments at the enterprise level to ensure that a balanced mix of project  and resource investments is maintained. Officials noted that excessively  focusing on segments that require new and innovative products can result  in long cycle times, wasted money, and lost opportunities elsewhere.  Likewise, critical opportunities can be lost when too much emphasis is  placed on simply continuing to invest in old markets with old products.  According to the officials we spoke with, the current investment mix for  Motorola’s Government and Enterprise Mobility Solutions business unit is  roughly 70-20-10, where 70 percent of its projects and resources are  dedicated to maintaining its core business, while 20 percent are invested  in pursuing new markets with existing products or introducing new or  enhanced products into existing markets, and the remaining 10 percent are  dedicated to discovering new markets and new products.
As part of their market analyses, companies increasingly refine their  understanding of who their customers are and what they need. For several  of the companies we met with, determining the needs of their customers is  complex because they have multiple groups of customers to consider. For  example, Eli Lilly has four customer groups with diverse needs: patients,  doctors, insurance companies, and government regulators. This  complexity is compounded when considering that success in a worldwide  market is critically dependent on a company’s ability to operate within  different governmental systems, laws, and regulations; and regional  markets. Several of the companies we reviewed use a variety of methods— including interviews, surveys, focus groups, and concept tests—to actively  engage their customers and help determine what they need. Some  companies also observe customer behaviors to identify unstated wants  and needs that if met—assuming corporate knowledge and resources  allow—could actually exceed customer expectations. While companies  actively seek customer input to identify products that show the most  promise and satisfy customer needs, customers generally do not identify  specific products to be developed.
section_title: Companies Follow a Disciplined Process to Identify New Products and Achieve a Balanced Portfolio
paragraphs: Once companies have identified and prioritized their market opportunities,  they follow a disciplined process to assess the costs, benefits, and risks of  potential product alternatives and allocate resources to achieve a balanced  portfolio that spreads risk across products, aligns with the company’s  strategic goals and objectives, and maximizes the company’s return on  investment. At an early stage, each alternative product is expected to be  accompanied by an initial business case that contains knowledge-based  information on strategic relevance and estimates of cost, technology  maturity, and the cycle time for getting the product from concept to  market. To ensure comparability across alternatives, companies require  initial business case information to be developed in a transparent manner,  to use specific standards, and to report estimates within certain levels of  confidence or allowable deviations. Each of the companies we reviewed  also stressed the importance of having multiple management review  points, or gates, at early phases to assess and prioritize alternative  products. As products move through review gates, from ideas, to more  concrete concepts, to the start of development where a final business case  is made, companies expect uncertainties—which are typically inherent in  the early phases—to be addressed and estimates to become more precise.  Consequently, the number of viable alternatives tends to decrease at each  review gate as those with the lowest potential for success and least value  are terminated or deferred, while those that are poised to succeed and  providing the best value are approved to proceed (illustrated in fig. 2).  Companies emphasized that making tough go/no-go decisions is critical to  keeping a balanced portfolio. Over time, as potential new products are  identified, companies review them against other product investments  (proposed and existing) and rebalance their portfolios based on those that  add the most value.
The companies we visited each follow a disciplined, gated review process  to ensure that they commit to development programs that help balance the  portfolio and that are executable given available corporate resources. This  allows companies to avoid committing to more programs than their  resources can support and ensure stability in the programs they invest in.  Although the number of review gates prior to the start of full-scale product  development varied between companies—ranging from four at Procter &  Gamble, to eight at Motorola—they all required potential products to  follow an established, disciplined process and meet specified criteria at  each review point. For example, Caterpillar assesses product alternatives  at four review gates prior to the start of development—three of which  were recently added to enhance the rigor of its investment decision  making. Each alternative must be supported by a draft business case that  includes quantifiable data that can be compared with specific standards  and used to determine if the related product can move past that gate. At  each gate, alternatives are reviewed to ensure that knowledge about  critical technologies, life-cycle costs, product reliability, and product  affordability is being acquired and that the product contributes to  achieving the company’s strategic goals and objectives.
Because developing a new drug is costly and time consuming, Eli Lilly  requires that the data supporting potential new drugs must meet high  standards to ensure that managers are informed to make sound investment  decisions. Each potential new drug must be supported by an initial  business case that contains information about safety and efficacy;  forecasted revenue; expected unit demand; capital, medical, supply and  material, development, and selling and marketing expenses; and general  administrative costs. The initial business case must also identify critical  success factors, state the probability of technical success, and provide a  timeline that details when major milestone events are expected and how  long it will take to get the associated drug to market. Eli Lilly assesses,  approves, and funds proposed new drugs incrementally. At each milestone  review a contract is established between the project team and a  gatekeeper committee, which contains deliverables, time frames, and the  costs to get to the next milestone. Once this contractual agreement is  reached the budget is allocated for the entire phase. The gatekeeper  committees expect each new drug proposal to achieve an 80-percent  confidence level in their cost and schedule estimates for the next phase.  This high level of confidence is achievable in large part because final  budget estimates are not developed by project teams until 2 months prior  to the milestone review. Projects are terminated at early points in the  review process when it is determined that their critical success factors  cannot be achieved. Because Eli Lilly’s projects typically have a high  degree of technical risk, only about 1 percent of those that start early  development actually make it to the marketplace.
Motorola officials also emphasized the importance of having sound  information when assessing potential new products. They noted that a  process without sound information will not produce good outcomes.  Therefore, Motorola’s Government and Enterprise Mobility Solutions  business unit expects potential products to be supported by initial  business cases containing data that meet specific standards and levels of  confidence at each review gate. For example, cost estimates for potential  products are developed in several phases and are expected to increase in  confidence with each successive phase. Early in the investment planning,  when an initial business case is first drafted, the confidence parameters  are generous, ranging from as much as 75 percent higher to as much as   25 percent lower than what the project will likely end up costing. By the  time a product alternative reaches the beginning of product development,  when a final business case is made, Motorola expects the cost estimates to  be at confidence levels of 10 percent higher and 5 percent lower. Proposed  products that fail to meet the specified criteria at early review gates are  either terminated or sent back to further mature and reenter the review  process from the beginning.
The companies we reviewed use a variety of portfolio management tools  and methods to inform the investment and resource allocation decisions  they make at each review gate. Some companies employ scoring methods,  using experts to rate products based on a number of factors—such as  strategic fit, risk, and economic value—and use that information to  prioritize alternative products. Another common tool plots alternative  products on a decision matrix that compares factors such as costs and  benefits, or risks and rewards of competing alternatives. Using this type of  matrix, alternative products are often represented by circles, where the  size of the circles provides information about key constraints such as  available annual resources or the estimated annual costs for each  alternative. For example, figure 4 compares risk and expected rewards by  plotting competing alternatives on a matrix. Alternatives that fall into the  upper left quadrant are high risk and low reward, while alternatives that  fall into the lower right quadrant are low risk and high reward. By  weighing risk against rewards and considering constraints such as annual  resources or annual cost, this tool provides critical information and a  structured means to help managers make informed decisions. Company  officials at Procter & Gamble emphasized the importance of selecting a  balanced mix of products to pursue. They noted that pursuing only low- risk and high-reward products at the expense of more innovative, higher- risk products could cause the company to miss out on opportunities to  improve their competitive standing in the marketplace. Likewise,  excessive pursuit of higher-risk products with the potential for high  returns could also result in lost opportunities to elsewhere.
Recognizing the inherent risks in pursuing a new development program— that overruns or underruns in one business case result in lost opportunity  to invest resources in another worthwhile project—IBM permits products  to deviate from their original business case estimates as long as the  deviation is within established limits. These limits are specified in a  contractual document resulting from negotiations between senior  management and project managers and signed at the beginning of product  development. Product development teams are expected to execute  according to the contract; if established thresholds are breached, action is  taken immediately to reassess the product within the context of the  portfolio and determine whether it is still a relevant and affordable  investment to pursue.
section_title: Successful Portfolio Management Requires Strong Governance with Committed Leadership, Empowered Decision Makers, and Effective Accountability
paragraphs: Successful portfolio management requires strong governance with  committed leadership that empowers portfolio managers to make  decisions about the best way to invest resources and holds those  managers accountable for the outcomes they achieve. The companies we  reviewed indicated that it is critical to have commitment from the top  leaders of the organization and recognition at all levels that what is best  for the company must be a priority, and not simply what is best for a  particular business unit or product line. In addition, the companies  emphasized that roles and responsibilities for implementing portfolio  management, including the designation of who is responsible for product  investment decisions and oversight, must be clearly defined. Because  portfolio managers are on the front line, the companies we reviewed  empower these managers to make product investment decisions and hold  them accountable for outcomes, not just for individual products but also  for the overall performance of their portfolios. To support their portfolio  managers, the companies encourage collaboration and communication,  including sharing bad news early. Several companies also emphasized the  importance of supporting their portfolio managers with cross-functional  teams, composed of representatives from the key functional areas within  the company—such as science and technology, marketing, engineering,  and finance—to ensure that they are adequately informed when making  investment decisions. To ensure accountability, companies often use  incentives and disincentives, including promotion and termination. We  have previously reported that high-performing organizations have  monetary and other rewards that clearly link employee knowledge, skills,  and contributions to achieving the organization’s goals and objectives.  These organizations underscore the importance of holding individuals  accountable and aligning performance expectations with organizational  goals and cascade those expectations down to lower levels. Companies  stressed that the transformation to portfolio management takes time and  requires not only process changes but cultural changes throughout the  company.
Eli Lilly emphasized that a key to making its portfolio management  process work is having a single committee with a high-level official in  charge responsible for making product investment decisions. Previously,  the company had a multi-layered committee structure in place, and  decisions were made based on reaching a consensus—an approach that  was viewed as cumbersome and lengthy. Eli Lilly also ensures  accountability by directly linking management and employee bonuses to  the overall success of the company. Individual employee performance  objectives are aligned with specific company objectives, such as meeting  budgetary goals, time frames, and data quality levels for a given project.  Achievement of individual employee objectives is measured periodically to  provide feedback to the employee. Eli Lilly officials stressed that having  the right performance metrics in place is important because ultimately you  get what you measure; therefore, be sure to measure the right things.
Motorola considers accountability to be the critical factor in making its  portfolio management process successful. In addition, Motorola’s culture  is not averse to reporting bad news to management. Project managers are  encouraged to report problems early so that they can be addressed before  they get out of control. Senior managers, however, are not intimately  involved in the day-to-day decision making for individual products. That  responsibility, in nearly every case, is delegated to the business unit  general manager. The general manager of a business unit is held  accountable for ensuring that the products within his unit succeed at all  levels. The general manager is responsible for holding product managers  accountable for the attainment of critical knowledge at key points and the  performance of their individual products overall. General managers and  product managers can be fired for not meeting objectives. Motorola  believes that if managers are held accountable for results, then they have  more desire to get it right.
section_title: Lacking an Integrated, Portfolio-Based Approach, DOD Has Too Many Programs Competing for Limited Resources
paragraphs: Although the military services fight together on the battlefield as a joint  force, they do not identify warfighting needs and make weapon system  investment decisions together. DOD has taken steps to identify warfighting  needs through a more joint requirements process, but the department’s  service-centric structure and fragmented decision-making processes are at  odds with the integrated, portfolio management approach used by  successful commercial companies to make enterprise-level investment  decisions. Consequently, DOD has less assurance that its weapon system  investment decisions address its most important warfighting needs and are  affordable in the context of its overall fiscal resources. In addition, DOD  commits to products earlier than the companies we reviewed and with far  less knowledge about their cost and feasibility. This leads to poor program  outcomes and funding instability, as the department attempts to fix  troubled programs by taking funds from others.
subsections: section_title: Service-centric Structure and Fragmented Decision- making Processes Impede DOD’s Ability to Prioritize Warfighting Needs
paragraphs: Although recent DOD policy emphasizes a more joint approach to  identifying and prioritizing warfighting needs, DOD’s service-centric  structure and fragmented decision-making processes hinder the policy’s  successful implementation. This policy, which introduced the JCIDS  process, calls for a wider range of stakeholders than before, including  more customer (i.e., combatant command) involvement; introduces new  methodologies intended to foster jointness; and groups warfighting needs  into eight functional areas based on warfighting capabilities—such as  netcentric, force application, and battlespace awareness—that cut across  the military services and defense agencies. The JCIDS process emphasizes  early attention to the fiscal implications of newly identified needs,  including identifying ways to pay for new capabilities by divesting the  department of lower priority or redundant capabilities. Despite these  provisions, assessments of warfighting needs continue to be driven by the  services and to be based on investment decision-making processes that do  not function together to ensure that DOD pursues needs that its resources  can support.
The military services identify warfighting needs individually, and  department-level organizations are not optimized to integrate the services’  results or evaluate their fiscal implications early on. Historically, this  approach has contributed to duplication in weapon systems and  equipment that does not interoperate. At the department level, Functional  Capability Boards oversee each of the eight functional areas, reviewing the  services’ assessments, and providing recommendations to the Joint  Requirements Oversight Council (JROC), which leads the JCIDS process.  However, defense experts and DOD officials report that the Functional  Capability Boards do not have the staff or analytical resources required to  effectively evaluate service assessments within the context of the broader  capability portfolio and assess whether the department can afford to  address a particular capability gap. Several recent studies have  recommended that DOD increase joint analytical resources for a less  stovepiped understanding of warfighting needs. In addition, the boards  lack the authority to allocate resources and to make or enforce decisions  to divest their capability area of existing programs to pay for new ones— authority successful companies provide to their portfolio managers.  Finally, some defense experts contend that the service ties of JROC’s  members—that is, the services’ Vice Chiefs and the Assistant Commandant  of the Marine Corps—reinforce service stovepipes. To better ensure a  more joint perspective, they recommend a more diverse JROC, with  representatives from other department-level organizations and the  combatant commands.
Resource allocation decisions are made through a separate process—the  Planning, Programming, Budgeting, and Execution system (PPBE)—which  hinders the department’s ability to weigh the relative costs, benefits, and  risks of investing in new weapon systems early on. Within the PPBE  system, the individual military services are responsible for budgeting and  allocating resources under authority that is commonly understood to be  based on Title 10 of the United States Code. PPBE is structured by  military service and defense program, although the department integrates  data on the services’ current and projected budget requests under   11 crosscutting mission areas called Major Force Programs. The cross- cutting view provided by the Major Force Program structure is intended to  facilitate a strategic basis for resource allocation, allowing the Secretary of  Defense to more easily see where the greatest mission needs are and to   re-allocate funds to meet those needs regardless of which service stands to  gain or lose. However, we have reported in the past that the Major Force  Program structure has not provided sufficient visibility in certain mission  areas. Moreover, although they cut across the services, the program  mission areas are not consistent with the more recently established  capability areas used in the JCIDS process, and as a result, it is difficult to  relate resources to capabilities. For example, in prior work, we observed  that the Major Force Programs contain large numbers of programs with  varied capabilities, complicating comparisons needed to understand  defense capabilities and associated trade-off decisions. We have  recommended that DOD report funding levels for defense capabilities in  its Future Years Defense Program report to the Congress, which is  currently organized by the Major Force Programs.
In addition, our analysis of DOD’s investment accounts—which pay for  developing, testing, and buying weapon systems and other equipment— indicates that DOD generally does not allocate resources on a strategic  basis. Figure 5 illustrates that the service allocations as a percentage of the  department’s overall investment budget have remained relatively static for  the 25-year period we examined, even though DOD’s strategic environment  and warfighting needs have changed dramatically during that time, with  the demise of the cold war and the emergence of the global war on terror.  In contrast, successful commercial companies using portfolio management  would expect to see their resource allocations across business areas to  reflect changes in the marketplace and the competitive environment.
PPBE and JCIDS are led by different organizations (see fig. 6), as is the  third of the three processes involved in DOD’s weapon system investment  decisions, the Defense Acquisition System (DAS), making it difficult to  hold any one person or organization accountable for investment  outcomes. The 2006 Quadrennial Defense Review highlighted the need for  governance reforms, and a 2006 study commissioned by DOD observed  that the budget, acquisition, and requirements processes are not  connected organizationally at any level below the Deputy Secretary of  Defense, concluding that this structure induces instability and erodes  accountability. The Under Secretary of Defense/Acquisitions,  Technology, and Logistics (USD/AT&L) has stated that weapon system  investment decisions are a shared responsibility, and, therefore, no one  individual is accountable for these decisions. At a broader, strategic level,  we have stated in prior work that DOD has lacked sustained leadership  and accountability for various department-wide management reform  efforts, including the establishment of an effective risk management  approach as a framework for decision making. This approach would link  strategic goals to plans and budgets, assess the value and risks of various  courses of action as a tool for setting investment priorities and allocating  resources at the department level, and use performance measures to  assess outcomes. To address the lack of sustained leadership, we have  supported legislation to create a chief management official at DOD.
OSD (Policy, PA&E, and Comptroller) responsibility Planning, Programming, Budgeting, and Execution System (calendar-driven)
JCS (JROC) — Joint Chief of Sff (Joint Requirement Overight Concil)
OSD (Policy, PA&E, nd Comptroller) — OSD (Policy, Progrm Anly nd Evuation, nd Comptroller)
OSD (AT&L) — OSD (Acquition, Technology nd Logitic)
The Office of the Secretary of Defense (OSD) does not assess the funding  implications of a proposed program at the front end of the investment  process, when it is initially validated by JROC. JCIDS is a continuous,  need-driven process that unfolds in response to warfighting needs as they  are identified. However, PPBE is a calendar-driven process comprised of  phases that occur over a 2-year cycle, thus OSD’s formal review of a  proposed program is not often synchronized with JROC’s, and can occur  several years later. Nevertheless, according to Joint Staff and AT&L  officials we met with, proposed programs begin to gain momentum when  they are validated by JROC, and they become very difficult to stop. These  officials indicated that momentum begins to gather because the services  start programming and budgeting for the proposed capability right away to  secure funding, generally several years before actual product development  begins and before OSD formally reviews the services’ programming and  budgeting proposals. In the interim, the services have not only budgeted  for their proposed programs, but established a program office, conducted  their Analysis of Alternatives, and identified specific user requirements.  OSD’s programming and budgeting review occurs at the back end of the  investment process, when it is difficult and disruptive to make changes,  such as terminating existing programs to pay for new, higher priority  programs.
These practices have contributed to the department starting more  programs than its resources can support. DOD defers much of the  additional cost of its programs into the future, resulting in what some have  characterized as a fiscal bow wave (illustrated in fig. 7). This bow wave  has grown at a pace that greatly exceeds DOD’s annual funding increases.  The cost remaining for DOD’s major weapons programs increased almost  135 percent between 1992 and 2006, while the department’s annual funding  level only increased 57 percent over that same time period. If this trend  goes unchecked, Congress will likely be faced with a difficult choice: pull  funds from other high-priority federal programs to support DOD’s  acquisitions or accept less warfighting capability than originally promised.
section_title: DOD Commits to a Solution Earlier and with Less Knowledge
paragraphs: DOD commits to a solution to address a warfighting need earlier in the  investment process than commercial companies do and before it has  adequate knowledge about cost and technical feasibility. Proposed options  for resolving a gap in military capability are submitted in an Initial  Capabilities Document (ICD). DOD guidance states that this document  should contain a range of approaches based in part on the cost and  technological feasibility posed by the approaches, laying the foundation  for a more detailed Analysis of Alternatives to be conducted under the  Defense Acquisition System. In addition, JROC is to receive a briefing on  the ICD that follows a standard format and addresses such issues as  linkage of the proposal to strategic guidance;  the time frame within which the capability is needed;  the threat/operational environment;  risks and assumptions (including the risk associated with  proceeding and not proceeding with solutions to each); and    a description of the best materiel and non-materiel approaches  based upon cost, efficacy, performance, technology maturity, and  risk.
Although DOD guidance calls for the analysis of a solution’s cost and  feasibility, we found that ICDs contained little of this type of information.  Several DOD officials we met with, who are directly involved in the JCIDS  process, did not believe cost and feasibility information was mandated at  this point. In our review of 14 unclassified ICDs approved by JROC from  2003-06, we found that 11 did not contain acquisition cost estimates and   12 did not contain estimates of the technical feasibility of proposed  solutions. We also found that JCIDS guidance does not specify the level  of accuracy sought in cost and feasibility estimates, and a white paper that  does provide recommendations in this regard is advisory.
We found that ICDs generally focused on the strategic, or operational,  relevance of proposed solutions, but a lack of guidance and an evolving  methodology have raised questions about the accuracy of data supporting  those assessments. JCIDS uses new joint warfighting concepts to  translate top-level military strategy into the capabilities a commander  might need on the battlefield. The joint concepts underpin a capabilities- based approach to identifying requirements, in which analyses are  expected to focus on broad military capabilities rather than service- specific platforms. However, the joint concepts and capabilities-based  assessments are works in progress. The concepts are being updated due to  concerns about their scope, and guidance on conducting a capabilities- based analysis has been lacking. Several DOD officials we met with stated  that assessments vary in their rigor, and a senior Joint Staff official said  that training on requirements development is one of three central  challenges at present. In January 2007, we reported that DOD officials  described concerns about the analytical framework for a capabilities- based assessment on joint seabasing, which could lead to inaccurately  identifying gaps in implementing the concept. Enhancing a seabasing  capability is expected to be costly and could be the source of billions of  dollars of investment if DOD chooses an option involving the development  of new ships.
DOD does not consistently follow a disciplined review process to ensure  that proposed solutions are making progress toward an executable  development program, although DOD policy emphasizes that such reviews  are necessary. DOD’s policy identifies several key decision points prior to  starting a new weapon system development program:     an initial decision point, where the Initial Capabilities Document is  reviewed, validated, and approved by the JROC;    a Concept Decision review, where entry into the concept refinement  phase of the Defense Acquisition System should be authorized; and     a Milestone A decision point, where a preferred solution and a  technology development strategy should be reviewed and  approved.
Since Initial Capabilities Documents generally do not contain information  on cost and technical feasibility, the JROC does not have a sufficient basis  for making go/no-go decisions at the initial decision point. In the 4 years  since JCIDS was implemented, nearly all of the warfighting needs  identified by the services and submitted for review in an ICD have been  validated and sent into the acquisition pipeline for further analysis as  potential programs, which calls into question whether go/no-go decisions  are the point of this first key gate. Information on cost and feasibility is  generally developed after the ICD is approved and proposed solutions  undergo further refinement through an Analysis of Alternatives (AOA). An  AOA should compare alternative solutions in terms of life-cycle cost,  schedule, and operational effectiveness, leading up to the identification of  a preferred alternative. However, officials from PA&E and the Joint Staff  indicate that AOAs often make a case for a single preferred solution.  Several of them indicated other concerns about AOAs, such as not setting  up trade-off discussions, lack of analytical rigor, length, and timeliness.
In any case, the next review points—the Concept Decision and Milestone  A—are often skipped; thus, the opportunity to review an evolving business  case and to make go/no-go decisions is bypassed. In prior work, we found  that 80 percent of the programs we reviewed entered the Defense  Acquisition System at Milestone B without holding any prior major  reviews, such as a Milestone A review. Such reviews are intended to  provide acquisition officials with an opportunity to assess whether  program officials had the knowledge needed to develop an executable  business case. Senior officials with OSD confirmed that this is a common  practice among defense acquisition programs. We concluded that this  practice eliminates a key opportunity for decision makers to assess the  early product knowledge needed to establish a business case that is based  on realistic cost, schedule, and performance expectations. In addition, we  found that programs are regularly approved to begin development even  though officials reported levels of knowledge below the criteria suggested  in DOD’s acquisition policy.
There is, then, generally little department-level oversight between the  point at which an ICD is approved and when system-level requirements are  validated and product development is initiated. At this point, as we  indicated earlier, there is generally no turning back, because the services  have invested considerable time and money, established a budget, and  formed a constituency for a proposed program, and decision makers  become reluctant to terminate a program or send it back for further study.
section_title: DOD Is Piloting Several Initiatives to Address Disconnects in Investment Decision-making
paragraphs: In response to the 2006 Quadrennial Defense Review and other recent  acquisition reform studies, DOD has undertaken several key, interrelated  initiatives intended to strengthen the department’s approach to investment  decisions. The initiatives include (1) taking a new approach to reviewing  proposed concepts that will provide decision makers with an early  opportunity to evaluate trade-offs among alternative approaches to  meeting a capability need, (2) testing portfolio management approaches in  selected capability areas to facilitate more strategic choices about how to  allocate resources across programs, and (3) using capital budgeting as a  potential means to stabilize program funding. While promising, these  initiatives do not fundamentally change DOD’s existing service-centric  framework for making weapon system investment decisions.
To address a perceived gap between DOD’s major decision-making  processes and provide a department-level means to assess potential  solutions (materiel and non-materiel) to fill a validated capability need,  DOD is testing a new approach to a Concept Decision review, which will  take place after a warfighting need is validated by the JROC. This new  approach is intended to focus attention on the affordability and feasibility  of potential solutions and generate early cost, schedule, and performance  trade-offs prior to the point of a significant investment commitment. As  currently proposed, the Concept Decision will be informed by a newly  required Evaluation of Alternatives that will integrate the Functional  Solutions Analysis conducted under JCIDS with the Analysis of  Alternatives conducted under the acquisition system and lay out the  relative merits and limitations of potential solutions. Furthermore, concept  decision reviews will be implemented by a tri-chair board consisting of  lead decision makers from the JCIDS, PPBE, and DAS processes. While  promising, the Concept Decision review largely reinstitutes a review point  that already existed but was only intermittingly used. For Concept  Decision reviews to be effective, DOD will have to establish enforcement  and accountability mechanisms to ensure the reviews are actually  implemented. In addition, the extent to which the concept reviews can  achieve desired effects will depend on what authority Concept Decisions  carry and who will be held accountable, particularly in light of the service- dominated investment structure that currently exists.
The department has also begun to pilot-test capability-based portfolio  management, selecting four joint capability areas to focus on—joint  command and control, joint net-centric operations, battlespace awareness,  and joint logistics. The intent is to enable the department to develop and  manage capabilities, as opposed to simply individual programs, and  enhance the integration and interoperability within and across sets of  capabilities. Each portfolio is being structured somewhat differently to  help the department determine how best to proceed with portfolio  management. All, however, are intended to focus initially on existing  programs and to operate within DOD’s existing decision-making  framework. The portfolios are largely advisory and will, as a first step,  provide input to decisions made through the JCIDS, PPBE, and DAS  processes. At this point, the capability portfolio managers have not been  given direct authority to manage fiscal resources and make investment  decisions. Without portfolios in which managers have authority and  control over resources, DOD is at risk of continuing to develop and  acquire systems in a stovepiped manner and of not knowing whether its  systems are being developed within available resources.
DOD is also examining the use of capital accounts as a potential means of  stabilizing program funding, which has long been cited as a significant  issue in program management. This capital budgeting pilot initiative is in  the early stages of planning, and the specifics of how such accounts will  be implemented are being developed, but the intent is for DOD to commit  a set amount of funding for the development portion of a project and hold  to that commitment by not adjusting funding up or down until the product  is delivered. In addition to resource constraints, programs would be given  a fixed amount of time to get from one milestone to the next. If successful,  this initiative could represent a step toward stabilizing long-term costs  within major defense acquisition programs, as well as a strengthening of  the ability of program managers to conduct long-term planning and control  costs. However, for this initiative to be effective, DOD will need to  overcome long-standing problems it has had in starting programs without  sufficient knowledge of the costs, requirements, and technologies needed  to develop proposed weapon systems. Unless this changes, it is unlikely  that capital accounts will lead to increased program stability.
section_title: Conclusions
paragraphs: While DOD has increasingly strengthened its ability to operate as a joint  force on the battlefield, the department’s organizational structures,  processes, and practices for planning and acquiring weapon systems are  not similarly joint. Put simply, DOD largely continues to base its  investment decisions on service-driven analyses that do not provide an  enterprise-level understanding of overall warfighting needs and on  individual platforms rather than broader sets of capabilities. In contrast,  successful commercial companies use an integrated portfolio management  approach to focus early investment decisions on products collectively at  an enterprise level and to ensure there is a sound basis to justify the  commitment of resources. By following a disciplined, integrated process— where the relative pros and cons of market opportunities and competing  product proposals are assessed based on available resources and  customer needs, and where tough decisions about which investments to  pursue are made—companies are able to reduce duplication between  business units, move away from organizational stovepipes, and effectively  support each new development program they commit to. Until DOD takes  a joint, portfolio management approach to weapon system acquisition— with functionally aligned entities that have the requisite responsibility,  authority, and control over resources—it will continue to struggle to  effectively prioritize warfighting needs, make informed trade-offs, and  achieve a balanced mix of weapon systems that are affordable, feasible,  and provide the best military value to the warfighter. Committing to more  programs than the budget can support and approving programs based on  insufficient knowledge to effectively manage risks will further delay  providing critical capabilities to the warfighter and lead to lost  opportunities to address other current and emerging needs.
section_title: Recommendations
paragraphs: We recommend that the Secretary of Defense implement an enterprise- wide portfolio management approach to making weapon system  investments that integrates the assessment and determination of  warfighting needs with available resources and cuts across the services by  functional or capability area. To ensure the success of such an approach,  the Secretary should establish a single point of accountability at the  department level with the authority, responsibility, and tools to ensure  that portfolio management for weapon system investments is effectively  implemented across the department.
In addition, the Secretary should ensure that the following commercial  best practices, identified in this report, are incorporated:  implement a review process in which needs and resources are  integrated early and in which resources are committed  incrementally based on the achievement of specific levels of  knowledge at established decision points;    prioritize programs based on the relative costs, benefits, and risks  of each investment to ensure a balanced portfolio;  require increasingly precise cost, schedule, and performance  information for each alternative that meets specified levels of  confidence and allowable deviations at each decision point leading  up to the start of product development;    establish portfolio managers who are empowered to prioritize  needs, make early go/no-go decisions about alternative solutions,  and allocate resources within fiscal constraints; and    hold officials at all levels accountable for achieving and maintaining  a balanced, joint portfolio of weapon system investments that meet  the needs of the warfighter within resource constraints.
We also recommend that the Secretary take steps to support department- level decision makers and portfolio managers by developing a stronger  joint analytical capability to assess and prioritize warfighting needs.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report. The  comments appear in appendix II.
DOD concurred with the majority of our recommendations and partially  concurred with two. Generally, in responding to these recommendations,  DOD stated that it is undertaking several initiatives and pilot efforts to  improve the department’s approach to investment and program decision  making, and that implementation of any new business rules will be  contingent upon the outcome of these initiatives. The department also  stated that it is experimenting with portfolio management, related  authorities and organizational constructs, and integrated decision-making  processes.
We believe that these initiatives and pilot efforts may be steps in the right  direction, but we are concerned that they do not go far enough to address  the systemic cultural and structural problems identified in this report.  DOD has attempted many similar acquisition reform efforts over the past 3  decades, including significant revisions to both defense requirements and  acquisition policy. However, despite these efforts, weapon system  acquisition programs continued to experience cost overruns, schedule  slips, and performance shortfalls. The department’s current initiatives are  likely to face the same fate because they do not fundamentally change  DOD’s service-centric framework or sufficiently integrate its decision- making processes for making weapon system investments.
DOD did not provide comments regarding our recommendation that the  Secretary establish a single point of accountability at the department level  with the authority, responsibility, and tools to ensure that portfolio  management for weapon system investments is effectively implemented  across the department. We believe that a single point of accountability is  necessary to successfully implement a portfolio management approach  and integrate DOD’s fragmented decision-making processes under one  senior official who is accountable for weapon system investment  outcomes. We further believe that our recommendations would better  position DOD to make tough, knowledge-based choices among potential  weapon system investments.
We are sending copies of this report to the Secretary of Defense; the  Secretaries of the Air Force, Army, and Navy; and the Director of the  Office of Management and Budget. We will provide copies to others on  request. This report will also be available at no charge on GAO’s Web site  at http://www.gao.gov.
If you have any questions about this report or need additional information,  please call me at (202) 512-4841 (sullivanm@gao.gov). Key contributors to  this report were John Oppenheim, Assistant Director; Lily Chin;   John Krump; Matthew Lea; Travis Masters; Sean Seales; Karen Sloan;  Susan Woodward; and Rebecca Yurman.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report examines the Department of Defense’s (DOD) requirements  identification and resource allocation processes for major weapons  systems. The primary focus is on identifying successful private-sector  principles and practices that could be adopted by DOD to help improve  stability in weapon system acquisition programs. Specifically, our  objectives were to (1) identify best practices of successful commercial  companies for ensuring that they pursue the right mix of programs to meet  the needs of their customers within resource constraints and (2) compare  DOD’s enterprise-level processes for investing in weapon systems to those  practices. Our work was conducted between March 2006 and February  2007, in accordance with generally accepted government auditing  standards.
We analyzed the outputs of DOD’s investment decision-making support  processes—the requirements determination process known as JCIDS and  the resource allocation process known as PPBE—using criteria  established in DOD policy and in previous GAO reports. We identified  impacts of the existing processes by analyzing quantitative and qualitative  data on DOD spending trends, conducting interviews with DOD officials,  and reviewing previous reports by GAO and by other knowledgeable audit  and research organizations. In addition, we met with officials representing  the Office of the Secretary of Defense, Joint Staff, and military services. At  each of these locations, we conducted interviews that helped us describe  the current condition of DOD’s requirements identification and resource  allocation processes. We also reviewed DOD and military service policies  and funding documents pertaining to the DOD requirements identification  process and resource allocation decisions for major weapons systems.  Specifically, we reviewed the contents of 14 unclassified Initial Capability  Documents that were finalized after June 24, 2003—the publication date  for the JCIDS instruction—to assess the extent to which they contained  cost and technical feasibility information. Those 14 ICDs were  unclassified, weapon system-related ACAT I, II, or III ICDs that were  contained in the Joint Staff requirements database. We relied on previous  GAO reports that highlight both the symptoms and causes of unstable  requirements and funding in DOD weapons acquisition programs. A list of  these reports can be found at the end of this report. In addition, we  reviewed recent key studies and reports addressing acquisition reform  issues by the Center for Strategic International Studies, Institute for  Defense Analysis, the U.S. Naval War College, the Defense Acquisition  Performance Assessment Project, the Joint Defense Capabilities Study  Team, the Joint C4ISR Decision Support Center, the Defense Science  Board, and the 2001 and 2005 Quadrennial Defense Reviews.
We also reviewed pertinent literature from authoritative corporate,  academic, and professional organizations, to identify commercial best  practices and processes that could be used by DOD to improve its weapon  system investment decision-making processes. In addition we conducted  case studies of five leading commercial companies. In selecting them, we  sought to identify companies that were recognized in the literature for best  practices, had large and diversified portfolios of products, and make  significant investments in the development and production of new  products. For each of the companies, we interviewed management  officials knowledgeable about their requirements identification and  resource allocation activities, to gather consistent information about  processes, practices, and metrics the companies use to help achieve  successful product development outcomes. Below are descriptions of the  five companies featured in this report:  Motorola is a Fortune 100 global communications leader that provides  seamless mobility products and solutions across broadband, embedded  systems, and wireless networks. According to Motorola’s 2005 Corporate  Profile, the company is the market leader in mission critical wireless  communication systems, two-way radios, embedded telematics systems,  digital set-top shipments, cable modem shipments, digital head-ends,  embedded computer systems for communication applications, CDMA  infrastructure sales (excluding the United States), and second in world  wide wireless handsets. Motorola achieved net sales of $31.323 billion and  spent $3.060 billion on research and development in 2004. The corporation  has approximately 68,000 employees, in 320 facilities, spanning   73 countries. We met with the management of Motorola’s Government &  Enterprise Mobility Solutions and Global Telecom Solutions sectors in  Schaumburg, Illinois.
International Business Machines (IBM)
IBM is one of the world’s largest technological companies, spending about  $3 billion annually on research and development activities. It is the largest  supplier of hardware, software, and information technology services. With  3,248 U.S. patents, IBM earned more patents than any other company for  the 12th consecutive year in 2004. In the past 4 years, IBM inventors  received more than 13,000 patents—approximately 5,400 more than any  other patent recipient. IBM has over 329,000 employees worldwide. We  met with managers from IBM Integrated Product Development (IPD) in  Somers, New York.
Procter & Gamble (P&G)
Procter & Gamble Corp. (P&G) is a leading producer of consumer goods.  It currently leads in global sales and marketshare among all fabric care,  baby care, feminine care, and hair care products. It currently has over  130,000 employees in 80+ countries. Twenty-two of its brands have annual  gross sales exceeding $1 billion each. In fiscal year 2005/2006, P&G  invested $2.075 billion or 3 percent of net sales in research and  development (R&D). This ranks them as one of the top 20 largest research  & development investors among U.S.-based companies. P&G has more  Ph.D.s working in labs around the world than the combined science and  engineering faculties of Harvard, MIT, and Berkeley. We met with the  management of P&G’s New Initiative Delivery team in Cincinnati, Ohio.
Eli Lilly is a global pharmaceutical company and one of the world’s largest  corporations. It was founded over 130 years ago and currently employs  approximately 42,000 people worldwide, including 13,991 employed at its  headquarters in Indianapolis, Ind. Approximately 8,336 employees   (19 percent of the total work force) are engaged in research and  development (R&D); clinical research is conducted in over 50 countries;  there are R&D facilities in 9 countries; and manufacturing plants in   13 countries. Its products are marketed in 143 countries. Lilly’s net sales in  2005 were $14.6 billion. Eli Lilly strives to grow sales by 6 percent to   7 percent each year. In 2005, $3 billion was spent on R&D, a $334.4 million  increase from the previous year. Currently, R&D represents 20.7 percent of  sales. Lilly’s total R&D investment in the last 5 years from continuing  operations was $12.5 billion. We met with managers from Eli Lilly’s  Corporate Headquarters in Indianapolis, Ind.
Caterpillar is a technology leader and the world’s leading manufacturer of  construction and mining equipment, diesel and natural gas engines, and  industrial gas turbines. In 2005, its total sales and revenues were   $36.3 billion, and its total R&D expenditures exceeded $1 billion,  compared with $20.5 billion sales and $696 million R&D in 2001. Between  2001 and 2005, the average return on equity of its stockholders’ shares  more than doubled. Caterpillar has over 85,000 employees, and over  105,000 people are employed by Caterpillar’s dealers worldwide. We met  with managers responsible for Caterpillar’s New Product Introduction  (NPI) process in Peoria, Illinois.
section_title: Appendix II: Comments from the Department of Defense
section_title: Related GAO Products
paragraphs: Best Practices: Stronger Practices Needed to Improve DOD Technology  Transition Processes. GAO-06-883. Washington, D.C.: September 14, 2006.
Defense Acquisitions: Major Weapon Systems Continue to Experience  Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368.  Washington, D.C.: April 13, 2006.
DOD Acquisition Outcomes: A Case for Change. GAO-06-257T.  Washington, D.C.: November 15, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to  Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393.  Washington, D.C.: March 1, 2004.
Best Practices: Setting Requirements Differently Could Reduce Weapon  Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February  11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15,  2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8,  2001.
Best Practices: A More Constructive Test Approach Is Key to Better  Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31,  2000.
Defense Acquisition: Employing Best Practices Can Shape Better  Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April  26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System  Programs Implement Best Practices. GAO/NSIAD-99-206. Washington,  D.C.: August 16, 1999.
Best Practices: Better Management of Technology Development Can  Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington,  D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program  Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible.   GAO/T-NSIAD-98-123. Washington, D.C.: March 17, 1998.
Best Practices: DOD Can Help Suppliers Contribute More to Weapon  System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires  Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:  February 24, 1998.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,  1996.