Acquisition Reform: DOD Should Streamline Its Decision-Making Process for Weapon Systems to Reduce Inefficiencies
Released: 2015-02-24T00:00:00

Letter

Background

As of December 2014, DOD’s portfolio of major defense acquisition programs included 78 programs with a total estimated acquisition cost of roughly $1.4 trillion. The Under Secretary of Defense for Acquisition, Technology and Logistics is the defense acquisition executive, and for 38 of these 78 programs, the Under Secretary is the milestone decision authority, responsible for making decisions at major program milestones. These programs are referred to as Acquisition Category (ACAT) ID programs. For the remaining 40 programs, most of which are in production, the Under Secretary has delegated milestone decision making authority to the cognizant military service acquisition executive; these programs are referred to as ACAT IC programs. DOD also has programs that have not entered the engineering and manufacturing development phase. These programs are not yet part of the portfolio, but are expected to enter soon. For these programs, the Under Secretary is normally the milestone decision authority.

In DOD’s acquisition process, weapon system programs typically proceed through three major milestones—A, B, and C—where program offices provide information to the milestone decision authority in order to make a decision on whether the program is ready to transition to the next acquisition phase. The milestones normally represent transition points in the overall acquisition process where there is a marked increase in the resources required for the program. Milestone A is the decision for an acquisition program to enter into the technology maturation and risk reduction phase; Milestone B is the decision to enter the engineering and manufacturing development phase; and Milestone C is the decision to enter the production and deployment phase. Figure 1 depicts DOD’s acquisition process.

DOD’s acquisition process is managed and supported by officials at different hierarchical levels. Weapon system program managers typically report to program executive officers in each military service who are charged with overseeing the execution of a portfolio of related systems such as fighter aircraft or ships. Program executive officers, in turn, typically report to a military service acquisition executive, who reports to the defense acquisition executive. As part of the milestone decision process, programs are reviewed at each level before reaching the milestone decision authority. Figure 2 shows the different levels.

Statutes and DOD policy require the documentation of specific information on major defense acquisition programs at each acquisition milestone. Our review focused on the information required at Milestone B, most of which is also expected at Milestone C. Appendix II includes a list and description of these information requirements. While several different Office of the Secretary of Defense organizations and other organizations have responsibility for compiling and documenting the information, the majority of the responsibility rests with the program office managing the acquisition.

For nearly 20 years, GAO has examined the best practices for product development from over 40 commercial firms to identify potential opportunities for DOD to adopt and implement those practices. For this review, we visited five leading commercial firms that follow a gated or milestone process in developing new products. While their business models are different than DOD’s, and often their products are less technically complex, commercial firms share a common goal with DOD in delivering their products to their customer on time and within cost estimates. Leading commercial firms can provide alternative approaches for milestone decision processes.

Information Required for Milestone Decisions Takes Significant Time and Effort to Document, but Does Not Always Add High Value

Programs we surveyed spent on average over 2 years completing the steps necessary to document up to 49 information requirements for their most recent acquisition milestone. This includes the time for the program office to develop the documentation and for various stakeholders to review and approve the documentation. These 49 information requirements also took, in total, on average 5,600 staff days for programs to document. However, on average, almost half of these requirements, 24 of the 49, were not highly valued by the acquisition officials we surveyed. Four major defense acquisition programs we examined illustrate the challenges in completing the milestone decision process.

Programs Spend Considerable Time and Resources Documenting the Information Required at Milestones, but the Majority Are Not Highly Valued By Acquisition Officials

Programs can spend a significant amount of time documenting up to 49 information requirements in advance of a Milestone B or C review. The requirements cover a vast array of program information, such as information on the overall acquisition strategy to justify the business case for a program; detailed implementation plans, such as those for systems engineering and testing; informational reports, analysis, and assessments; and decisions and certifications.

We surveyed 24 program managers that held a milestone B or C decision since 2010 and found that it took them over 2 years on average to complete the entire set of documents needed for the milestone decision. The program managers, as well as other acquisition officials we surveyed, considered on average about half of the information requirements as not highly valued. Figure 3 provides a summary of this information. More details about the survey results are presented in appendix III.

Programs spent an average of about 1 year to complete each information requirement. However, as shown in figure 3, there was a wide range in the length of time it took to complete documentation, as some took almost 2 years to complete and some took less than 6 months. About half of the time for each information requirement was spent documenting the information and the other half for review. These 49 requirements also took, in total, on average 5,600 staff days for programs to document. We did not ask programs to provide data on the staff days needed to review and approve the documentation because they do not have access to data on the amount of time officials at levels above them spend completing this process.

As shown in figure 3, acquisition officials on average considered

24 requirements as providing high value to their organization’s role in the milestone decision process,

20 requirements as providing moderate value, and

5 requirements as providing less than moderate value.

Information requirements considered high value by stakeholders include a program’s acquisition strategy, sustainment plan, and information related to planned technologies, cost, and testing. Several senior acquisition officials we met with considered many of these requirements as critical to the program’s business case,the capabilities required of the weapon system, the strategy for acquiring the weapon system, and the cost, schedule, and performance baselines. which typically includes documentation on Information requirements valued the least (less than moderate value), on the other hand, include such documentation as the benefit analysis and determination for potentially bundling contract requirements; the Clinger- Cohen certification for information technology investments; the corrosion prevention control plan to assess the impact of corrosion on cost, availability, and safety of equipment; the item unique identification implementation plan for managing assets; and the replaced system sustainment plan for documenting the estimated cost to sustain a system until the new program is fielded. One service acquisition executive, for example, stated that program managers should not have to develop an item unique identification implementation plan because government contractors put the unique identification numbers on parts. Another senior official stated that the Clinger-Cohen Act requirements are geared towards the acquisition environment of the 1990s. This official believes the requirements should be updated to reflect the current environment for procuring information systems.

As part of the process of documenting the information required at the milestones, program officials brief cognizant officials responsible for the different functional areas, such as test or systems engineering, as well as senior leadership and the milestone decision authority on specific aspects of the program’s overall plans. The briefings, done in parallel with the actual process of documenting the required information, are used as a forum for DOD to discuss the information and to determine a program’s readiness for the milestone decision. Program offices can spend a great deal of time and effort briefing the different officials and senior leaders in advance of the milestone decision. Data provided by 9 of the programs we surveyed that recently had a milestone B decision showed that programs provided an average of 55 briefings over a period of just over a year and a half leading up to the milestone.

Four Acquisition Programs Illustrate the Significant Time and Effort to Complete the Information Requirements

We examined four major defense acquisition programs, at least one from each military service that recently held a milestone decision, to get more specific details of the time and effort expended by programs to complete the milestone decision process. All four programs needed about 24 months to complete the process. While the number of documents varied for each program, it took an average of over 13 months to complete each document based on three programs that could provide data. Two of these programs used contractors to provide assistance in completing the documents. Figure 4 provides a summary of the overall effort required of these four programs—two of which were preparing for milestone B and the other two were preparing for milestone C.

Two of the programs that tracked the staff days required to prepare the milestone documents told us they spent 3,800 and 9,867 staff days, respectively. These same programs also used contractors to assist with the documents.

The Number of Stakeholder Reviews at Each Milestone Is a Large Contributor to the Effort Needed to Complete Information Requirements

A primary reason it takes over 2 years to complete the information required for a milestone decision is the large number of stakeholders that review the documents at the many organizational levels above the program office. We found that stakeholders in many different offices among 8 different levels can review the information and documentation needed to support a milestone decision. According to the program offices we surveyed, these reviews added only moderate or less value to most documents. DOD recognizes that it has too many levels of review and has several initiatives to eliminate the acknowledged bureaucracy, but has had limited success implementing changes to reduce the time and effort needed to review documentation.

Reviews Are Conducted at Multiple Levels by Different Functional Organizations

The information and documentation required at milestones can be reviewed by as many as eight different organizational levels before a decision is reached on whether a program is ready for the next acquisition phase. In general, the information is reviewed at each level to gain approval before the program provides the information to the next level. This is done serially, which takes more time. Eventually, the defense acquisition executive and other senior executives review the information and determine whether the program is ready to proceed to the next acquisition phase. Figure 5 shows the multiple levels of reviews.

Many different functional organizations within each level review the information before the document is approved. The number of organizations conducting reviews varies depending on the information included in each document. A few documents that include a wide breadth of information can be reviewed by many offices at each level. For example, Air Force acquisition strategies, that on average took over 12 months to complete for the programs we surveyed, can be reviewed by 56 offices, some more than once, before being approved. Figure 6 lists the organizations involved in this review process.

The reviews of more narrowly focused documentation also go through different levels, but may be reviewed by fewer organizations at each level. As one example, offices at as many as four levels took an average of 7 months to review a program’s Technology Readiness Assessment, based on responses to our survey. This assessment is prepared prior to Milestone B to show the results of an assessment of the maturity levels of the critical technologies planned to be integrated onto the program. Initially, the program office prepares an assessment of the different technologies’ maturity levels, taking into account the conclusions reached by a panel of independent subject matter experts. Then, the program executive officer reviews and approves the assessment. Next, a service level expert, with possible assistance from a science and technology expert, reviews the assessment. After that, officials from the Office of the Secretary of Defense, Assistant Secretary for Research and Engineering office evaluate the assessment and make their own independent assessment. Finally, the milestone decision authority certifies whether to approve the program to enter engineering and manufacturing development or defer this decision until technologies are mature. Each of these four levels of review, done serially, can present new questions and comments that need to be resolved before the program can satisfy the information requirement.

GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, GAO-08-467SP (Washington, D.C.: Mar. 31, 2008); and Defense Acquisitions: Assessments of Major Weapon Programs, GAO-04-248 (Washington, D.C.: Mar. 31, 2004). program results is related instead to incentives. We have reported previously on several factors that create incentives for DOD to deviate from sound acquisition practices and reform initiatives. These factors include (1) mismatches between capability requirements and the knowledge, funding, and time planned to develop a new system, (2) programs being started to fill voids in military capability but quickly evolving to address other, conflicting demands, and (3) programs being funded in a way where there are few consequences if funding is not used efficiently.

DOD has recognized that its extensive review process is a challenge. A DOD study in 2011 highlighted the many organizational levels of oversight and said DOD has a “checkers checking checkers” system, which contributes to inefficiencies that can undermine program managers’ execution of programs because they spend too much time complying with the oversight process, including documenting the information requirements.time and resources addressing conflicting comments/concerns expressed by the functional offices at the different levels during the review process. Officials also told us the functional staff conducting reviews typically wanted significantly more information than their superiors want or need and this often leads to multiple revisions. For example, the Deputy Assistant Secretary for Systems Engineering has indicated he wants limited, specific information in a systems engineering plan and even issued guidance to promulgate this direction. Despite this direction, we were told the systems engineering plan for one Navy program grew from 100 pages to 243 pages in length, because staff wanted additional information added as it went through the review process. In contrast, however, one of the three service acquisition executives we surveyed and some senior level officials within the Office of the Secretary of Defense stated that staff reviews are helpful as they ensure the documentation is sufficient before executives at each level perform their review.

Several program officials told us they spend extensive Service officials also told us that while it is important to get input from functional staffs on their areas of expertise, these staffs can have “tunnel vision,” or focus only on their respective area and do not adequately consider whether their recommended changes to documentation might add schedule time, additional costs, or have other effects on a program. Officials expressed frustration that functional staffs are not held accountable for the potential effect on a program as a result of their recommended changes. Recently, the Under Secretary of Defense for Acquisition, Technology and Logistics has tried to clarify the role of some staff, stating in a memorandum that the service acquisition executives, the program executive officers, and program managers are responsible and accountable for the programs they manage; everyone else (i.e., staff supporting the Office of the Secretary of Defense staff) has a supporting or advisory role.

Documentation Reviews Are Not Highly Valued by Program Officials

While there are multiple levels and many organizations involved in reviews, overall the 24 program managers we surveyed did not think these reviews added significant value to the documentation. The program managers considered the value added to 10 percent of the documentation to be high. However, for the remaining 90 percent of the documents, the officials believed the reviews did not add high value—61 percent were moderate and 29 percent less than moderate. Figure 7 provides a summary of the program offices’ assessment.

Of the 14 documentation reviews that were considered to add less than moderate value, 2 documents were reviewed for an average of 10 months each and the others ranged between 2.5 and 8.5 months. Other service level officials we surveyed—program executive officers and the three service acquisition executives—had views similar to the program managers; they considered the value added to be high for less than 10 percent of the documentation.

DOD Has Had Limited Success in Streamlining Its Milestone Decision Process

DOD has acknowledged that too much time is spent on reviews and preparing documents and has taken some steps over the past several years to address some of the unproductive steps identified in its milestone decision processes. For the most part, however, efforts to date have been limited in scope and have not had a significant effect on the amount of time and effort program offices spend on documentation required at milestones. One has even stalled. Examples of these efforts include: In 2011, the Under Secretary for Acquisition, Technology and Logistics delegated the approval authority for three milestone documents from the Office of the Secretary of Defense level to the service level. This reduced the number of levels of review and reviewers of these documents. A DOD official told us the approval authority for additional documents could be delegated in the future, but no additional documents are currently being considered.

In 2013, the Under Secretary for Acquisition, Technology and Logistics asked the service acquisition executives to identify programs where the milestone decision authority could be potentially delegated from the Office of the Secretary of Defense to a lower level. Delegation to the lower level also reduces the number of levels of review and reviewers. The services identified 18 programs: 7 programs from the Air Force; 5 programs from the Navy; and 6 programs from the Army. In September 2014 the Under Secretary of Defense for Acquisition, Technology and Logistics delegated the authority to act as the milestone decision authority to the Secretary of the Air Force for 3 programs, the Secretary of the Navy for 1 program, and the Secretary of the Army for 1 program.

In April 2013, the Under Secretary of Defense for Acquisition, Technology and Logistics issued guidance that included a potential pilot test of a “skunkworks” process for major defense acquisition programs. The Under Secretary requested that each service recommend one candidate program for a pilot test by July 2013. As of October 2014, programs had not been identified and the effort has been placed on hold. Office of the Secretary of Defense officials stated it has been difficult to identify programs that meet the Under Secretary’s expected preconditions—namely to identify programs that have well defined requirements, a strong relationship with industry, and a highly qualified and appropriately staffed government team that can remain with the program until it is delivered.

In 2014, DOD began using an Electronic Coordination Tool designed to electronically disseminate and track the progress of documentation being reviewed. The tool is used to enforce time limits for the review of documents and provide near real-time views of all comments made during the review process to promote greater efficiency across the department. DOD officials have begun using this tool with the Acquisition Strategy and hope to add more documents over time.

DOD is currently assessing many of the documents it develops in response to statutory information requirements and plans to propose legislative modifications to Congress in the spring of 2015 to help streamline documentation while still meeting the intentions of the statutes.

DOD’s revised acquisition policy has also placed greater emphasis on “tailoring,” which means modifying the traditional acquisition process, including documentation and reviews, to best suit a program’s needs.

However, a few program officials told us that trying to tailor by obtaining waivers for milestone requirements involves significant time and effort, and that it is often easier to simply complete the requirements rather than try to obtain waivers. While we did not examine the overall use of tailoring by DOD programs during our review, we examined two programs that attempted tailored documentation and reviews, but in the end, neither was able to make significant changes. Specifically, the F-22 Increment 3.2B program told us they requested waivers for 17 requirements, but ultimately only 2 were waived. In addition, the Long Range Strike– Bomber, in direction provided by the former Secretary of Defense, was to be managed with a streamlined approach. The program was initially allowed the flexibility to tailor many of the needed documents and reviews. However, over time, these flexibilities have been scaled back.

DOD Has Experience Using a Streamlined Process, and Commercial Firms Also Offer Alternative Approaches

DOD has proven it can streamline its process. Several past programs, like the F-16 and F-117, were managed successfully with a more streamlined approach and DOD is currently using a more streamlined milestone decision process for some classified programs. Commercial companies we examined—Boeing, Caterpillar, Cummins, Honda, and Motorola Solutions—also use processes that minimize the levels of review resulting in a quicker, more efficient milestone decision process.

DOD’s Initial Acquisition Policy Provided a Framework for a More Streamlined Process

In 1971, DOD issued its initial 5000 acquisition policy. The policy, which totaled seven pages, provided for minimum formal reporting and more streamlined layers of authority than the complex process in place today. Specifically, the original guidance provided for (1) minimal layers of authority above the program office; (2) few demands on programs for formal reporting; (3) minimal demands for non-recurring information and for responding to these requests informally; and (4) the development of a single, key document to support program management and milestone decision making. Over time, a large, bureaucratic process has supplanted these elements. For example, requirements have been added to improve cost estimating, logistics planning, design reviews, and technology maturity assessments. Each of these areas has been in great need of improvement and individual documentation and review requirements were aimed at addressing known shortfalls.

Several studies by acquisition experts over the past decade have highlighted the need for DOD to again streamline its process. For example, a Defense Acquisition Performance Panel stated in its 2006 report that complex acquisition processes do not promote program success, but increase cost, add to schedules, and obfuscate accountability. The Panel recommended that DOD create a streamlined acquisition organization with accountability assigned and enforced at each level. In 2009, the Defense Science Board reported that DOD’s milestone decision process should take a few days of preparation, not the months and months currently required. The report describes a process with too much bureaucracy, overlap and diffusion of responsibilities, and a need for excessive coordination among acquisition organizations. The report recommended that DOD streamline the acquisition process.

DOD Has Used a More Streamlined Process with Success

The F-16 program, developed in the 1970s, was managed under a streamlined process laid out in the early acquisition guidance. DOD officials have often stated that one contributing factor to the F-16 program’s success was the use of a more streamlined approach, where the number of levels of review and reviewers was minimized, and emphasis was placed on real-time program reviews in lieu of preparing formal reports and documents. According to one former DOD senior official, the program office was staffed with an experienced program manager and functional experts, which worked closely and collaborated with functional offices to achieve a common goal of fielding a usable combat capability as quickly as possible. The F-16 program also operated with different incentives than most programs, which enabled a more streamlined approach. For example, the F-16 was developed as a low- cost fighter with a strategy that involved making incremental technology improvements and incorporating performance trades by the customer to keep costs down.

The F-117 aircraft, which was largely developed in the early 1980s in a classified security environment, was managed with a “skunkworks” approach. According to a RAND study, central to the F-117 program approach was its flexibility and responsiveness in decision-making.DOD leadership delegated more decision-making to the program office, with an associated reduction in detailed, document oversight by higher levels. RAND stated that the willingness to delegate decision making authority to lower levels enabled a quicker response to problems. A former Air Force senior leader during the program’s development, who later served as the Under Secretary of Defense (Acquisition and Technology), stated that the program held monthly meetings between the functional managers and program management. The meeting participants were empowered to make decisions and did not need to seek approval from their senior leadership after the meetings. It was expected that any issues would be addressed with their leadership prior to the meetings. The frequent interactions reduced the need for reports, documents, and reviews. In another report, a former senior Air Force acquisition officer, who also served as program director for the F-117, reported that the ability to have a quicker process comes from pushing decision-making to the lowest levels without having to proceed up the chain of command for approval to implement decisions.

DOD is using a more streamlined approach for some of its current classified programs that may have the potential to make the milestone decision process more efficient. A few classified programs we reviewed are managed with a process that includes fewer levels and reviewers between the program office and decision authority. the program manager reports to the program executive officer who reports directly to a Board of Directors comprised of the service acquisition executive, service secretary and chief, and the defense acquisition executive. The Board of Directors serves as the milestone decision authority for the programs. Decisions by the Board are unanimous agreements by all members. Leading up to the Board meeting, programs have separate, focused interactions with a small number of key functional offices as necessary. According to service officials, establishing this short, narrow chain of command allows for a more expedited decision-making process that requires less time and resources. Figure 8 shows the levels of review for the milestone decision process.

Section 2430 of title 10, U.S. Code, specifically excludes highly sensitive classified programs (as determined by the Secretary of Defense) from the definition of a major defense acquisition programs. Therefore, statutes governing major defense acquisition programs generally do not apply to classified programs.

Commercial Firms Provide Alternative Approaches for Milestone Decisions

Commercial companies we examined—Boeing, Caterpillar, Cummins, Honda, and Motorola Solutions—use a more streamlined process than DOD traditionally uses for its major defense acquisition programs. Companies prepare similar documents as DOD acquisition programs, but only a few of the most critical ones, the business case documents, require senior management approval. A key enabler to this approach is the establishment of frequent, regular interactions between program officials and decision makers. Companies minimize the levels of review needed to determine whether a program is ready to advance to the next acquisition phase, resulting in a quicker, more efficient process.

The companies prepared documents similar to those of DOD such as development, test, engineering, and manufacturing plans. Officials at Motorola Solutions, Cummins, and Boeing stated that most documents are prepared and approved by functional managers assigned to the program office core team. Programs prepare an integrated document that summarizes key program information for the decision makers to review and approve. Figure 9 illustrates the levels at which documents are generally prepared and approved for commercial companies we visited.

According to company officials, the integrated document may include information on customer requirements, resources, schedules, risks, technical data, and market launch plans. As part of the milestone decision process, program managers also provide evidence that the other program documents have been completed and approved by the appropriate official.

For the companies we visited, ensuring that the program management team has a strong link to decision makers was a critical factor to their streamlined approach. Several companies held meetings between program officials and senior managers at frequent intervals to assess progress towards the next milestone decision. Officials stated that frequent, regular interactions enable senior managers to stay informed of program issues and plans, allowing the decision meeting to focus on making a well-informed decision, instead of spending time bringing decision makers up to date (see fig. 10).

Cummins functional managers, for example, meet one-on-one with senior program managers on a monthly basis to review program progress. Officials stated that about 2 weeks prior to the decision meeting, a comprehensive review is conducted with each program’s functional area manager, supporting team, and senior functional manager to ensure required activities have been completed before a milestone review and the plan going forward is sound.

Boeing officials stated that they conduct a series of monthly meetings between program functional area managers and senior managers to assess whether a program is meeting the criteria needed for moving into the next phase. According to officials, the results provide support for the milestone decision.

Honda has established an environment that encourages frequent, direct interaction between program participants. Senior managers, program managers, and staff work in an open bullpen environment, rather than offices. This layout facilitates real-time discussions across organizational levels, multiple programs, and functional areas. Issues can be quickly discussed and resolved as they arise so only the most important ones need to be addressed at the milestone decision meetings.

Companies we visited told us it typically takes only a few months or sometimes even a few weeks to complete the milestone decision process. The process for these companies included one or two levels of review to assess whether a program is ready to advance to the next phase. For example, Motorola Solutions and Cummins use a process in which programs proceed directly to the decision maker after they have packaged together information needed to support a milestone decision. Motorola Solutions program officials provided information to their decision makers about a week in advance of the decision meeting. During that week, program officials meet individually with principal members on the decision-making committee. The purpose of these meetings is not to present the program’s plans but to address any last minute concerns. According to Motorola officials, the decision meeting typically lasts about 30 minutes because issues are usually resolved in these earlier meetings. Boeing and Honda generally include one additional level of review.

The commercial model, in which good program outcomes can be achieved with a more streamlined oversight process, includes a natural incentive that engenders efficient business practices. Market imperatives incentivize commercial stakeholders to keep a program on track to meet business goals. In addition, awards and incentives for managers are often tied to the company’s overall financial success. As a result, commercial managers are incentivized to raise issues early and seek help if needed. They know if the program fails, everyone involved fails because market opportunity is missed and business revenues will be impacted. Commercial product development cycle times are relatively short (less than 5 years), making it easier to minimize management turnover and to maintain accountability. DOD’s acquisitions occur in a different environment in which cycle times are long (10 to 15 years), management turnover is frequent, accountability is elusive, and cost and schedules are not constrained by market forces. Seen in this light, DOD must have an oversight process that substitutes discipline for commercial market incentives. Several industry officials stated that companies often add oversight levels or reviews as a first reaction after failures or problems occur. However, the officials further stated that this does not solve the root problems and often it makes the process less efficient. Two companies we visited highlighted an inspection-intensive oversight process they implemented as a deliberate attempt to address problems that had occurred but found that it led to an adversarial environment and an inefficient process. Both companies eventually abandoned this approach and replaced it with an approach where program officials are incentivized to reach out to recognized experts within the company for assistance when needed.

Conclusions

Over time, DOD has essentially tried to overcome a legacy of negative cost and schedule weapon system program outcomes by requiring extensive documentation to support program strategies, plans, and other information prior to a milestone decision. Much of the information required in this documentation was added by policy as well as statute and these requirements likely represented legitimate reactions to problems. However, the consequence of this approach is that an extensive process has built up, in which program offices and other DOD organizations spend an enormous amount of time and effort preparing and reviewing documentation. Given the persistence of weapon system acquisition problems over decades, especially schedule delays and cost overruns, the effort involved with documenting and reviewing information requirements does not appear to correspond to the value gained. Programs we surveyed spent over 2 years completing information requirements that in some instances can be reviewed by as many as 56 organizations at eight levels. In the end, program officials felt almost half of these information requirements were not of high value. Further, program managers did not highly value the reviews by higher level DOD organizations for 90 percent of the documentation.

The need to document information about essential aspects of a program and for an appropriate level of review and approval is legitimate. However, over time, the outcomes of weapon system programs have proven resistant to the oversight process. At the same time, the process has become bloated, time-consuming, and cumbersome to complete. The challenge is to find the right balance between having an effective oversight process and the competing demands such a process places on program management. Meeting the challenge will depend on DOD’s ability to identify the key problem areas in weapon system acquisitions and the associated root causes that exist today and whether information requirements and reviews are linked to addressing these problems. As we have noted in prior work, the most important information requirements—those that enable a program to establish a sound business case—include well-defined requirements, reasonable life-cycle cost estimates, and a knowledge-based acquisition plan. If information requirements and reviews are not clearly linked with the elements of a sound business case and/or the key issues facing acquisitions today, then they can be streamlined or even eliminated. If they are linked, but are not working well, then they warrant re-thinking. While the data support that change is needed, change does not mean weakening oversight, as unsatisfactory outcomes from acquisition programs may persist. Rather, the goal of change is to perform effective oversight more efficiently, and to recognize problems or incentives that require remedies and not just more information requirements.

In this time of decreasing defense budgets, where every dollar spent on inefficient activities is one less dollar available for modernizing our future force, a close look at the review process is warranted to provide stakeholders needed information in a more efficient and cost effective manner. The surveys of DOD acquisition officials we conducted, the results of which are shown in figure 3 and appendix III, highlight information requirements that provide less than moderate value to acquisition officials. These requirements, as well as ones that take a year or more to complete, could serve as a starting point for discussions on what documentation is really needed for weapon acquisition programs and how to streamline the review process. Officials within the Office of the Secretary of Defense believe the Electronic Coordination Tool shows promise for reducing review times on documents. Currently, it is being used to reduce review times for acquisition strategies, and other documents may be added in the future. Automating the document review process, however, is relatively easy compared to potentially eliminating levels of review because that will require DOD to move away from its “checkers checking checkers” culture and make tough choices as to which levels of review do not add value and are not necessary. If DOD does not eliminate levels of review, inefficiencies are likely to continue. According to federal internal control standards, agencies should develop effective and efficient processes to ensure that actions are taken to address requirements, such as in this case, completing the required information to aid in milestone decisions. In other words, DOD should be striving to make its process more efficient.

Selecting pilot test programs to experiment with streamlined acquisition processes, while capturing lessons learned from the pilot, would be steps in the right direction. The pilot programs could rely on practices used by some DOD classified programs and private industry companies we visited—namely, fewer information requirements and levels of review and more frequent interaction between the program office and actual decision makers.

Recommendations for Executive Action

To help improve DOD’s milestone decision process, we recommend that the Secretary of Defense direct the Under Secretary of Defense for Acquisition, Technology and Logistics in collaboration with the military service acquisition executives, program executive officers, and program managers take the following two actions: In the near term, identify and potentially eliminate (1) reviews associated with information requirements, with a specific focus on reducing review levels that do not add value, and (2) information requirements that do not add value and are no longer needed. For the remaining reviews and information requirements, evaluate and determine different approaches, such as consolidating information requirements and delegating approval authority, which could provide for a more efficient milestone process. This effort should also include a re-examination of the reason(s) why an information requirement was originally considered necessary in order to determine what information is still needed and if a more efficient approach could be used. Findings and survey responses included in this report could be used as a starting point for this examination.

As a longer-term effort, select several current or new major defense acquisition programs to pilot, on a broader scale, different approaches for streamlining the entire milestone decision process, with the results evaluated and reported for potential wider use. The pilot programs should consider the following:

Defining the appropriate information needed to support milestone decisions while still ensuring program accountability and oversight. The information should be based on the business case principles needed for well-informed milestone decisions including well defined requirements, reasonable life-cycle cost estimates, and a knowledge-based acquisition plan.

Developing an efficient process for providing this information to the milestone decision authority by (1) minimizing any reviews between the program office and the different functional staff offices within each chain of command level and (2) establishing frequent, regular interaction between the program office and milestone decision makers, in lieu of documentation reviews, to help expedite the process.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. DOD concurred with both of our recommendations. DOD’s comments are reprinted in appendix IV.

DOD concurred with our first recommendation, indicating that the Department’s Better Buying Power initiative contains efforts to streamline documentation requirements and staff reviews and its recent (February 2015) set of legislative proposals to Congress seeks to reduce some DOD reporting requirements. We acknowledge these efforts as steps in the right direction. We believe DOD can and should do more to eliminate reviews and information requirements that do not add value and are no longer needed. For the most part, efforts to date have been limited in scope and have not yet had a significant impact on the amount of time and effort program offices spend on documentation required at milestones. The Under Secretary of Defense for Acquisition, Technology and Logistics acknowledged in April 2014 that DOD has not had significant success in eliminating unproductive processes and bureaucracy. We also note that DOD’s recent set of legislative proposals to Congress for inclusion into the National Defense Authorization Act for Fiscal Year 2016 primarily seek to reduce reporting requirements, but do not address streamlining the many levels of review. As we reported, a primary reason it takes over 2 years to complete the information required for a milestone decision is the large number of stakeholders that review the documents at the many organizational levels above the program office. While it will take a coordinated effort on the part of the Department, we believe DOD can reduce the many levels of review.

DOD also concurred with our second recommendation, indicating that while not yet fully implemented, it has a Better Buying Power initiative to identify appropriate programs to pilot test a streamlined acquisition approach. As we reported, however, DOD has not yet identified candidate programs even though the initiative was proposed in April 2013 and was supposed to begin in July 2013. DOD officials told us it has been difficult to identify programs that meet the preconditions for the pilot set by the Under Secretary of Defense for Acquisition, Technology and Logistics— namely programs that have well defined requirements, a strong relationship with industry, and a highly qualified government team that can remain with the program until it is delivered. We encourage DOD to initiate the pilot, specifically on some current programs that have recently held a milestone B or will be approaching this milestone soon (e.g. Presidential Helicopter, Armored Multi-Purpose Vehicle, Joint Air-to- Ground Missile, Next Generation Jammer, Amphibious Combat Vehicle), as long as the aforementioned criterion of well defined requirements is considered. Almost two years have passed since the initiative was first proposed and even after DOD decides on the specific programs for the pilot, it will most likely be several years until lessons learned can be documented and potentially applied to other programs. We reiterate that, when implemented, each pilot should examine different approaches for streamlining the entire milestone decision process, including defining the appropriate information needed to support milestone decisions, such as business case principles like well defined requirements, reasonable life- cycle cost estimates, and a knowledge-based acquisition plan. Pilot programs should also strive to develop a more efficient process for providing this information to the milestone decision authority, which would most likely include minimizing reviews between the program office and the different functional staff offices within each chain of command level and establishing frequent, regular interaction between the program office and milestone decision makers, in lieu of documentation reviews.

We are sending copies of this report to interested congressional committees; the Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology and Logistics; and the Secretaries of the Air Force, Army, and Navy. This report also is available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on the matters covered in this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

This report examines the Department of Defense’s (DOD) weapon system acquisition process. Specifically we examined (1) the effort and value involved in DOD’s preparation for a milestone decision, (2) the factors that influence the time needed to complete the milestone decision process, and (3) alternative processes used by some DOD programs and leading commercial firms.

To determine the value and effort involved in DOD’s preparation for a milestone decision, we collected data from current and future major defense acquisition programs.

First, we distributed two questionnaires by email in an attached Microsoft Word file asking program managers of current major defense acquisition programs to (1) provide a value for each information requirement applicable at either Milestone B or Milestone C; (2) provide a value for the review of each information requirement for either Milestone B or Milestone C; (3) provide the length of time required to develop each information requirement; (4) provide the number of staff days spent by the program office to develop each information requirement; (5) provide the length of time it took each information requirement to get through the review and approval process; and (6) identify the primary users and customers of each information requirement. One questionnaire was sent to 11 program managers of current major defense acquisition programs identified in the Defense Acquisition Management Information Retrieval system as having completed a Milestone B decision review since January 1, 2010, and a different questionnaire to 15 program managers of current major defense acquisition programs identified in the Defense Acquisition Management Information Retrieval system as having completed a Milestone C decision review since January 1, 2010. We received responses from 24 program managers, between July and October 2014, 11 program managers from Milestone B programs and 13 program managers from Milestone C programs. Because there is a slight variation in the number of information requirements applicable at Milestone B verses at Milestone C, in our analysis we excluded 2 information requirements applicable at Milestone C—the Capability Production Document and the General Equipment Valuation.

We took a number of steps to ensure reliability of the data collected through our questionnaires, including reviewing responses to identify obvious errors or inconsistencies and conducting follow-up to clarify responses when needed.

Second, in a separate data collection effort to determine the number of briefings and the length of time needed to complete the milestone decision process, we submitted questions for an electronic questionnaire distributed to 55 programs, as part of GAO’s Annual Weapons System Assessment. We asked programs if they had completed a milestone decision review as of January 1, 2011, and if yes, to provide additional information regarding that milestone review. Twenty-four programs out of 55 responded that they had completed a milestone decision review in that time frame; however, not all programs provided information on the review. Four programs were excluded from our analysis because they were unable to provide the additional data. Another 5 programs were excluded because we determined they have been designated as an Acquisition Category IC program; our analysis was limited to Acquisition Category ID programs. Of the 15 programs in our analysis, 11 are current programs and 4 are future programs. Our results are not intended to be generalizable and as such, results from nongeneralizable samples cannot be used to make inferences about all major defense acquisition programs.

To better understand DOD’s milestone process, we selected 4 major defense acquisition programs to use as case studies to gain more in- depth knowledge about the milestone decision process: the Air Force’s F- 22 Increment 3.2B Modernization program, the Navy’s P-8A program; and the Army’s Joint Light Tactical Vehicle and Paladin Integrated Management programs. We used a data collection instrument to ensure we received similar information for all 4 case study programs in our review. We collected data on the number of briefings the program office held with program executive officers, service-level officials, and Office of the Secretary of Defense-level officials; the number of documents the program completed for the milestone decision review; and a timeline of their milestone decision review. In addition, we asked programs to provide detailed information related to the information requirements they prepared for the milestone, including the length of time spent documenting each information requirement; length of time it took the documentation to make it through the review process; the number of staff days the program office spent documenting each information requirement; and the cost to document each information requirement. We also reviewed milestone documents that programs prepared for the milestone in order to better understand what information is contained within the documents. Finally, we met with program officials from each case study program to obtain additional information on the milestone decision process.

We selected our case studies based on input from officials with the military services using the criterion that the program had been through either Milestone B or Milestone C since January 1, 2010. Further, the programs we selected for review represent each of the military services. Two programs—F-22 Increment 3.2B Modernization and Joint Light Tactical Vehicle—completed a Milestone B review and 2 programs—P-8A and Paladin Integrated Management—completed a Milestone C review. While our sample of four case studies allowed us to learn about inefficiencies with the milestone decision process, it was designed to provide anecdotal information, not findings that would be representative of all of the department’s major defense acquisition programs.

To determine the factors that influence the time needed to complete the milestone decision process, we met with officials and functional leaders and reviewed documents from several organizations within the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, including the Under Secretary. Specifically, we met with officials from the offices of (1) Acquisition Resources and Analysis; (2) Defense Procurement and Acquisition Policy; (3) Deputy Assistant Secretary of Defense for Systems Engineering; and (4) Deputy Assistant Secretary of Defense for Developmental Test and Evaluation. We also met with officials from the Office of the Director, Cost Assessment and Program Evaluation; and the Office of the Director, Operational Test and Evaluation.

We also met with officials and reviewed documents from the military services, including the Department of the Air Force, Department of the Army, and the Department of the Navy, including the service acquisition executives. Within each military service, we also met with officials from functional offices including the (1) Air Force Director of Test and Evaluation; (2) Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering; (3) Deputy Assistant Secretary of the Navy for Research, Development, Test & Evaluation; (4) Deputy Under Secretary of the Army for Test and Evaluation; and the (5) Director of Army System of Systems Engineering and Integration.

In order to capture the views of officials at the different levels involved in the milestone decision process, we also sent a questionnaire to program executive officers with responsibility for defense acquisition programs, all 3 military service acquisition executives, and 13 Office of the Secretary of Defense organizations identified as key stakeholders in the acquisition milestone decision process. We received responses from 25 program executive officers, all 3 military service acquisition executives, and 12 Office of the Secretary of Defense organizations.

We analyzed the data provided by program managers, program executive officers, military service acquisition executives, and Office of the Secretary of Defense officials to determine the overall value of the milestone information requirements and the overall value of the review of the information requirements to the various groups involved in the milestone decision process. Our results are not intended to be generalizable and as such, results from nongeneralizable samples cannot be used to make inferences about all major defense acquisition programs.

Further, we reviewed relevant statutes, DOD policies, and military service guidance for DOD acquisitions.

To examine alternative processes used by some DOD programs, we reviewed the processes used by some current classified programs. We also reviewed reports and studies done by acquisition experts that examined past programs, including the F-117 and F-16, which successfully used a more streamlined process. In addition, we examined acquisition policies that were in place at the time of these programs development.

To identify practices used by leading commercial firms that might be used to improve DOD’s acquisition process, we visited five companies to learn more about how they manage their product development processes. We selected these companies, in part, based on our previous GAO best practices work. These companies are recognized leaders in their industry, and are recognized for having successful, proven product development processes.

The companies selected for use in our review include:

Boeing, a leading aerospace company and a manufacturer of commercial jetliners. We met with officials and discussed their practices for managing the development of commercial aircraft in Seattle, Washington.

Caterpillar Inc. (Caterpillar), a leading manufacturer of construction and mining equipment, diesel and natural gas engines, and industrial gas turbines. We met with officials in Peoria, Illinois.

Cummins Inc. (Cummins), a leading manufacturer of diesel and natural gas-powered engines for on-highway and off-highway use. We met with officials in Columbus, Indiana.

Honda of America Manufacturing, Inc. (Honda), a leading manufacturer of motorcycles and automobiles. We met with officials at their location in Raymond, Ohio.

Motorola Solutions, a leading manufacturer of data capture devices such as professional and commercial radios and communication systems. We met with officials in Schaumburg, Illinois.

At each company we discussed the new product development process from concept to full production; the methods, tools, measures and metrics used by leadership in monitoring and overseeing product development execution progress; and roles and responsibilities of the product development manager.

We conducted this performance audit from January 2014 to February 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Information Requirements for Major Defense Acquisition Programs at Milestone B and C Identified in Interim DOD Instruction 5000.02

Enclosure 1 of the Interim Department of Defense (DOD) Instruction 5000.02 identifies several information requirements which must be documented for the milestones of the DOD acquisition process, as well as the source of each requirement in statute, a DOD directive, instruction, and/or manual, or a regulation. Not all of the information requirements are applicable at every milestone and not all of the requirements equate to a separate document.

Our review focused on documentation related to 49 statutory and policy information requirements that is expected to be completed at Milestone B, most of which is also expected at Milestone C. Of the 49 requirements applicable at Milestone B, 44 are also applicable at Milestone C. Two other information requirements are only applicable at Milestone C and not at Milestone B. We did not include these two requirements in our review, but they are listed below for a total of 51 requirements. The number of documents a program has to complete will vary depending on the type of program. For example, space programs have to complete an Orbital Debris Mitigation Risk Report, which is not required for non-space programs, and some requirements apply only to programs acquiring information technology.

Description Memorandum reflecting the milestone decision authority’s certification, prior to granting milestone approval, as to certain program matters.

Documents the decisions and direction resulting from each milestone and other major decision point reviews.

Agreement between the milestone decision authority and the program manager and his/her acquisition chain of command that will be used for tracking and reporting for the life of a program or program increment; contains schedule, performance, and cost parameters that are the basis for satisfying an identified mission need.

Describes the overall strategy for managing an acquisition program, including the program manager’s plan to achieve programmatic goals and summarizes the program planning and resulting program structure.

Provides a design constraint on the product DOD will build, procure, and sustain based upon the budgets DOD expects to have for the product over its life cycle.

Summarizes an analytical comparison of the operational effectiveness, suitability, and life- cycle cost (or total ownership cost, if applicable) of alternatives that satisfy established capability needs.

Documents the bandwidth requirements needed to support a program and how they will be met.

For bundled acquisitions, an analysis to determine the relative benefit to the government among two or more alternative procurement strategies and a determination of whether consolidation of the requirements is necessary and justified.

Describes how to redesign the way work is done to improve performance in meeting the organization’s mission while reducing costs.

Defines authoritative, measurable, and testable parameters across one or more increments of a material capability solution, by setting key performance parameters, key system attributes, and additional performance attributes necessary for the acquisition community to design and propose systems and to establish programmatic baselines.

Provides authoritative, testable capability requirements, in terms of key performance parameters, key system attributes, and additional performance attributes, for the production and deployment phase of an acquisition program.

For programs that acquire IT, documents compliance with the various requirements of the Clinger-Cohen Act of 1996, subtitle III of title 40, U.S. Code.

For programs containing information technology, documents a program’s plan for ensuring cybersecurity.

Promotes, monitors, and evaluates programs for the communication and exchange of technological data among defense research facilities, combatant commands, and other organizations involved in developing technological requirements for new items.

Documents the contract type selected by the milestone decision authority for a major defense acquisition program that is consistent with the level of program risk.

Ensures that opportunities to conduct cooperative research and development projects are considered at an early point during the formal development review process by indicating whether or not a project similar to the one under consideration by DOD is in development or production by another country or organization.

Description Determination of whether the weapon system or military equipment being acquired is necessary to enable the armed forces to fulfill the strategic and contingency plans prepared by the Chairman of the Joint Chiefs of Staff. If the determination is positive, then an estimate of those core capability requirements and sustaining workloads are provided, organized by work breakdown structure and expressed in direct labor hours.

Documents the plan to prevent and control corrosion from impacting the availability, cost, and safety of military equipment.

Cost Analysis Requirements Description Describes formally an acquisition program for purposes of preparing both the DOD Component Cost Estimate and the cost assessment and program evaluation independent cost estimate.

Cost analysis to support the Development Request for Proposal Release (RFP) decision point, which will vary depending on the program and information needed to support the decision to release the RFP.

Cost analysis conducted by the service cost agency.

Cost position established by the DOD component that is derived from the DOD Component Cost Estimate and the program office estimate per DOD component policy, and signed by the DOD component Deputy Assistant Secretary for Cost and Economics.

System-specific criteria which normally track progress in important technical, schedule, or management risk areas.

For systems that use the electromagnetic spectrum while operating in the United States and its possessions, a certification by the National Telecommunications and Information Administration (NTIA) that a candidate system conforms to the spectrum allocation scheme of the United States and its possessions.

Certifies that the DOD component will fully fund the program to the DOD Component Cost Position (CCP) in the current Future Years Defense Program (FYDP), or will commit to full funding of the CCP during the preparation of the next FYDP, with identification of specific offsets to address any funding shortfalls that may exist in the current FYDP.

Program description identifying contract-deliverable military equipment, non-military equipment, and other deliverable items and plans to ensure that all deliverable equipment requiring capitalization is serially identified and valued.

Cost estimate covering the full life-cycle cost of a program including all costs of development, procurement, military construction, and operations and support, without regard to funding source or management control, prepared or approved by the Director of Cost Assessment and Program Evaluation.

Analysis of a program’s supportability planning that assesses the program office’s product support strategy and how this strategy leads to successfully operating a system at an affordable cost.

Analysis that the skills and knowledge, processes, facilities, and equipment necessary to design, develop, manufacture, repair, and support a program are available and affordable.

Documents a program’s information-related needs in support of the operational and functional capabilities that the program either delivers or contributes.

Documents a program’s strategy to identify and manage the full spectrum of intellectual property (IP) and related issues throughout the program’s life-cycle describing, at a minimum, how the program will assess program needs for, and acquire competitively whenever possible, the IP deliverables and associated license rights necessary for competitive and affordable acquisition and sustainment over the product life-cycle.

Description Documents the program manager’s and product support manager’s plan for implementing item unique identification (IUID) as an integral activity within MIL-STD-130N item identification processes to identify and track applicable major end items and configuration- controlled items.

For programs that are dependent on intelligence mission data, defines specific intelligence mission data requirements for a program and becomes more detailed as the system progresses towards initial operational capability.

A living document describing a program manager’s approach and resources necessary to develop and integrate sustainment requirements into the systems design, development, testing and evaluation, fielding and operations.

Documents the quantity of the product needed to provide production representative test articles for operational test and evaluation and efficient ramp up to full production.

Provides out-year projections of active-duty and reserve end-strength, civilian full-time equivalents, and contractor support work-years for a major defense acquisition program.

Provides information on, among other things, whether there are commercial off-the-shelf products that meet the defined requirements in the business case, could be modified to meet requirements, or could meet requirements when it is necessary to modify those requirements to a reasonable extent.

Describes the operational tasks, events, durations, frequency and environment in which the materiel solution is expected to perform each mission and each phase of the mission.

Assessment of debris generation risk during launch, on-orbit operations, and end-of-life disposal, and compliance with the U.S. Government Orbital Debris Mitigation Standard Practices.

Documents the comprehensive approach to system security engineering analysis and the associated results to ensure that programs adequately protect their technology, components, and information throughout the acquisition process during design, development, delivery and sustainment.

Describes the strategy for integrating environment, safety, and occupational health considerations into the systems engineering process, how they are managed, and how they are integrated with human systems integration efforts.

For a program that will replace another program, documents the budget estimates required to sustain the existing system until the new system assumes responsibility; the milestone schedule for developing and fielding the new system; and an analysis of the ability of the existing system to maintain mission capability against relevant threats.

Communicates government requirements to prospective contractors and solicits proposals; defines the government’s expectations in terms of the performance and functional specifications, program planning, program process, risks, and assumptions; and reflects the program’s plans articulated in the draft Acquisition Strategy and other draft, key planning documents such as the Systems Engineering Plan, Program Protection Plan, Test and Evaluation Master Plan, and Life-Cycle Sustainment Plan.

Documents stretch goals for costs that DOD expects its leaders to do their best to reach, which are based on real opportunities, but challenging to execute.

Documents the program manager’s plan for the use of small business innovation research and small business technology transfer program technologies and associated planned funding profile.

Information requirement Spectrum Supportability Risk Assessment For spectrum-dependent systems, identifies and mitigates regulatory, technical, and operational spectrum supportability risks.

Documents key technical risks, processes, resources, metrics, systems engineering products, and completed and scheduled system engineering activities to help the program manager develop, communicate, and manage the overall systems engineering approach that guides all technical activities of a program.

Addresses projected adversary capabilities at system initial operating capability (IOC) and IOC plus 10 years; should be system specific, to the degree that the system definition is available at the time the assessment is being prepared.

Assesses the maturity of, and the risk associated with, critical technologies, to assist in the determination of whether the technologies of a program have acceptable levels of risk, based in part on the degree to which they have been demonstrated, and to support risk- mitigation plans.

Estimate of DOD’s potential liability if it terminates a contract for a program; the estimate must include how such termination liability is likely to increase or decrease over the period of performance.

Documents the overall structure and objectives of the test and evaluation program. It provides a framework within which to generate detailed test and evaluation plans and documents schedule and resource implications associated with the test and evaluation program.

Appendix III: Value of Information Requirements Assessed by DOD Organizational Levels

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Acknowledgments

GAO Contact

Michael J. Sullivan, (202) 512-4841 or sullivanm@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Cheryl K. Andrew, Assistant Director; Don M. Springman, Analyst-in-Charge; Julie C. Hadley; Matthew B. Lea; Brian T. Smith; Kristine R. Hassinger; Kenneth E. Patton; Laura S. Greifner; Oziel A. Trevino; and Nathaniel O. Vaught made key contributions to this report.