DOD Information Technology: Software and Systems Process Improvement Programs Vary in Use of Best Practices
Published: 2001-03-30T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: Several Department of Defense (DOD) components have software and systems process improvement (SPI) programs that are aligned closely to the best practices embodied in the Software Engineering Institute (SEI) IDEAL model and thus provide excellent examples of SPI. Elsewhere in DOD, however, such programs are lacking. Where they do exist, these programs are being credited with producing higher quality software and systems products faster and at less expense, whether managed in a centralized or decentralized fashion. The Office of the Secretary of Defense (OSD) has an important leadership role to play in expanding SPI across the department. In particular, it can seize opportunities to build upon and leverage the existing base of SPI programs within DOD's components and help ensure that all of its components realize the strategic value (i.e., benefits that exceed costs) that both private and public-sector organizations, including some DOD components, attribute to these programs. Although OSD faces funding choices among competing leadership initiatives, such as its efforts to conduct software acquisition maturity assessments and collect software metrics, these are some of the very tasks that are embedded within an effective SPI program. Thus, by ensuring that DOD components have effective SPI programs, OSD can leverage programs to indirectly accomplish its other high-priority initiatives as well.

section_title: Letter
paragraphs: With an annual information technology budget of about $20 billion, and  tens of billions more budgeted for technology embedded in sophisticated  weaponry, the Department of Defense (DOD) relies heavily on software- intensive systems to support military operations and associated business  functions, such as logistics, personnel, and financial management. One  important determinant of the quality of these systems, and thus DOD’s  mission performance, is the quality of the processes used to develop,  acquire, and engineer them. Recognizing the importance of these processes  to producing systems that perform as intended and meet cost and schedule  commitments, successful public and private organizations have adopted  and implemented software/systems process improvement (SPI) programs.
This report is part of our response to your request to compare and contrast  DOD information technology practices with leading practices. In particular,  you asked us to review DOD components’ (military services and Defense  agencies) SPI management activities to ensure that DOD is taking the  necessary steps to continuously strengthen its software and systems  development, acquisition, and engineering processes. As agreed with your  offices, our objectives were to (1) compare selected DOD components’ SPI  programs against Carnegie Mellon University’s Software Engineering  Institute’s (SEI) IDEALSM3 model, which is a recognized best practices  model, (2) determine how these components have approached  management of their SPI programs, and (3) determine what DOD-wide  efforts are under way to promote and leverage the components’ SPI  programs. The components that we selected were the Departments of the  Army, Air Force, and Navy; the Marine Corps; the Defense Logistics Agency  (DLA); and the Defense Finance and Accounting Service (DFAS).
Because Army, Navy, and Air Force do not manage SPI centrally and have  delegated SPI responsibility to their respective subordinate organizational  units, we selected at least two of the largest of these units within each  service to review. Accordingly, all references in this report to the respective  services’ SPI programs refer only to the subordinate units that we  reviewed. We performed our work from March through December 2000, in  accordance with generally accepted government auditing standards. (See  appendix I for details of our objectives, scope, and methodology, including  the specific service units reviewed.) DOD provided us with written  comments on a draft of this report. These comments are summarized in the  “Agency Comments and Our Evaluation” section of this letter and are  reproduced in full in appendix II.
subsections: section_title: Background
paragraphs: IDEALSM is a service mark of Carnegie Mellon University and stands for initiating,  diagnosing, establishing, acting, and leveraging. weapons systems, command and control systems, satellite systems,  inventory management systems, financial systems, personnel systems,  payment systems, and others. Many of these systems in turn are connected  with systems operated by private contractors, other government agencies,  and international organizations.
DOD’s ability to effectively manage information technology is critical to its  ability to accomplish its mission. Its reliance on software-intensive systems  to support operations related to intelligence, surveillance, security, and  sophisticated weaponry—along with financial management and other  business functions—will only increase as the department modernizes and  responds to changes in traditional concepts of warfighting.
The scope of DOD’s information technology inventory is vast: over 1.5  million computers, 28,000 systems, and 10,000 computer networks. Further,  many of DOD’s most important technology projects continue to cost more  than projected, take longer to produce, and deliver less than promised. As  a result, we have designated DOD systems development and modernization  efforts as a high-risk area.
The quality of the processes involved in developing, acquiring, and  engineering software and systems has a significant effect on the quality of  the resulting products. Accordingly, process improvement programs can  increase product quality and decrease product costs. Public and private  organizations have reported significant returns on investment through such  process improvement programs. SEI has published reports of benefits  realized through process improvement programs. For example, SEI  reported in 1995 that a major defense contractor implemented a process  improvement program in 1988 and by 1995 had reduced its rework costs  from about 40 percent of project cost to about 10 percent, increased staff  productivity by about 170 percent, and reduced defects by about 75  percent. According to a 1999 SEI report, a software development  contractor reduced its average deviation from estimated schedule time  from 112 percent to 5 percent between 1988 and 1996. During the same  period, SEI reported that this contractor reduced its average deviation  from estimated cost from 87 percent to minus 4 percent.
To aid organizations attempting to initiate and manage SPI programs, SEI  has published a best practices model called IDEAL,SM which defines a  systematic, five-phase, continuous process improvement approach, with a  concurrent sixth element addressing the program management tasks  spanning the five phases (see figure 1).
Initiating: During this phase, an organization establishes the  management structure of the process improvement program, defines  and assigns roles and responsibilities, allocates initial resources,  develops a plan to guide the organization through the first three phases  of the program, and obtains management approval and funding. Two key  organizational components of the program management structure  established during this phase are a management steering group and a  software engineering process group (SEPG). Responsibility for this  phase rests with senior management.
Diagnosing: During this phase, the SEPG appraises the current level of  process maturity to establish a baseline capability against which to  measure progress and identifies any existing process improvement  initiatives. The SEPG then uses the baseline to identify weaknesses and  target process improvement activities. It also compares these targeted  activities with any ongoing process improvement activities and  reconciles any differences. Responsibility for this phase rests primarily  with line managers and practitioners.
Establishing: During this phase, the SEPG prioritizes the process  improvement activities and develops strategies for pursuing them. It  then develops a process improvement action plan that details the  activities and strategies and includes measurable goals for the activities  and metrics for monitoring progress against goals. Also during this  phase, the resources needed to implement the plan are committed and  training is provided for technical working groups, who will be  responsible for developing and testing new or improved processes.  Responsibility for this phase resides primarily with line managers and  practitioners.
Acting: During this phase, the technical working groups, formed under  the establishing phase, create and evaluate new and improved  processes. Evaluation of the processes is based on pilot tests that are  formally planned and executed. If the tests are successful, the working  groups develop plans for organization-wide adoption and  institutionalization, and once approved, execute them. Responsibility  for this phase resides primarily with line managers and practitioners.
Leveraging: During this phase, results and lessons learned from earlier  phases are assessed and applied, as appropriate, to enhance the  structures and plans of process improvement programs. Responsibility  for this phase rests primarily with senior management.
The model’s sixth element, continuous program management, specifies  management structures and tasks for planning, organizing, directing,  staffing, and monitoring the program. Responsibility for this element rests  with senior management.
Each phase of the IDEALSM model contains several recommended tasks.  Appendix I, which describes our objectives, scope, and methodology,  identifies all tasks for each phase.
section_title: Components’ SPI Program Alignment With SEI IDEALSM Model Varies
paragraphs: The Army and Air Force units that we reviewed, as well as DFAS and two of  the four Navy units, have long-standing SPI programs that satisfy almost  every task recommended in the IDEALSM model (see table 1 for a summary  of how each component and its units, if applicable, compared to the  model). For example, in 1996 the Secretary of the Army mandated that all  software development, acquisition, and maintenance activities establish  SPI programs. Further, the Army requires that its software activities  continually improve their process maturity and has set maturity goals for  all of its units. Army regulations also mandate that contractors be  evaluated for software process maturity. Moreover, the two specific units  within the Army that we reviewed have SPI management structures, plans,  and dedicated resources. In addition, these units have continuously  evolved in software and system process maturity through many years of  assessing their baseline process capabilities, implementing new and  improved process initiatives, reassessing process maturity, and  implementing lessons learned. Both Army units satisfy all IDEALSM tasks.
In contrast, DLA, the Marine Corps, and two of the Navy’s four units that  we reviewed do not perform important IDEALSM model tasks. In particular,  DLA currently does not satisfy any of the model’s recommended tasks.  According to DLA officials, it had an SPI program prior to 1998, but at that  time the program was terminated to reduce costs. During our review, DLA’s  CIO stated that the agency plans to begin a new SPI program and has taken  a first step by assigning organizational responsibility.
The Marine Corps has many SPI activities under way that could form the  foundation of a program. However, it is not performing several key SPI  tasks that are fundamental to SPI program success. For example, the  Marine Corps has assigned responsibility for process improvement, and it  has begun assessing its software process maturity to establish baseline  capability. However, it is not using this baseline as a basis for implementing  recommended improvements, nor does it have an SPI plan or dedicated  resources for these activities. As such, the likelihood of the Marine Corps’  process improvement initiatives producing desired results is diminished.
Two of the four Navy software/systems units that we reviewed also do not  have SPI programs that are aligned with the IDEALSM model. To their credit,  however, one has recently taken the first step toward initiating a program  and the other has activities under way that could form the beginnings of a  program. (See appendix IV for more detailed results on each of the  components that we reviewed.)
section_title: Components’ SPI Management Approaches Vary, Yet All Report Positive Program Results
paragraphs: The four components that have SPI programs—Army, Air Force, DFAS, and  parts of the Navy—have different approaches for directing and controlling  their respective programs, ranging from centralized to highly decentralized;  each, however, reports positive results. For example, DFAS has a  centralized approach, with its headquarters office directing and controlling  all SPI activities. In contrast, the Army, Air Force, and Navy have  decentralized approaches to SPI program management. The Army, which  began its SPI program centrally, has since delegated SPI responsibility to its  commands, which—in the case of the two commands we reviewed—have  further delegated SPI program management to their respective  software/systems units. Similarly, the Air Force units that we reviewed  further delegated SPI management to their respective software/systems  units. The Navy commands follow different approaches—one manages its  program centrally and the other has delegated SPI management to its  software/systems units.
Despite different approaches, each DOD component/unit with an SPI  program reports positive effects on software/systems quality. DFAS, for  example, reports that its SPI program has reduced its cost to deliver  software to about one-third less than organizations of similar size. One  Navy software activity reports reduced costs, improved product quality,  and a 7:1 return on its SPI investment. An Army activity reports that it has  almost doubled its productivity in writing software for new systems  because of improvements made under its SPI program. (See appendix IV  for more detailed information on the approaches and reported benefits of  the components that we reviewed.)
section_title: DOD-Wide Efforts to Promote and Leverage SPI Programs Do Not Exist
paragraphs: Within OSD, the Assistant Secretary for Command, Control,  Communications, and Intelligence is responsible for establishing and  implementing DOD’s policies, processes, programs, and standards  governing the development, acquisition, and operation of nonweapons  systems software and information systems. Similarly, the Under Secretary  for Acquisition, Technology, and Logistics is responsible for establishing  DOD acquisition policies and procedures. Accordingly, OSD has an  important leadership role to play in ensuring that DOD components reap  the maximum possible benefits of effective SPI programs. Such leadership  can include dissemination of policies and guidance promoting SPI  programs and activities, knowledge of the nature and extent of  components’ SPI programs and activities, associated lessons learned and  best practices, and facilitation of SPI knowledge-sharing across DOD  components.
Both OSD organizational units have efforts under way aimed at improving  some aspects of DOD’s ability to develop and acquire software and  systems. For example, they have established teams to conduct software  acquisition maturity assessments and established a software collaborators  group. They also are collecting software metrics and establishing training  for managers.
However, OSD has no SPI actions under way or planned, such as issuing  policy and guidance on SPI programs; determining where in DOD SPI  programs do and do not exist; promoting the establishment of programs in  component units, such as DLA, where they do not exist; and sharing  knowledge across DOD about the experiences of reportedly successful SPI  programs, such as those within the Army, Air Force, DFAS, and parts of the  Navy. According to OSD officials, uncertainty about the costs versus  benefits of SPI, resource constraints, and other priorities have precluded  such a focus. However, as stated earlier in this report, various  organizations, including some DOD components, report positive returns on  investments from SPI programs that argue for SPI being treated as a  funding priority.
section_title: Conclusions
paragraphs: Several DOD components have SPI programs that are aligned closely to the  best practices embodied in the SEI IDEALSM model and thus provide  excellent examples of SPI. However, such programs are lacking in other  parts of the department. Where they exist, these programs are being  credited with producing higher quality software and systems products  faster and at less expense, whether managed in a centralized or  decentralized fashion.
OSD has an important leadership role to play in expanding SPI across the  department. In particular, it can seize opportunities to build upon and  leverage the existing base of SPI programs within DOD’s components and  help ensure that all of its components realize the strategic value (i.e.,  benefits that exceed costs) that both private and public-sector  organizations, including some DOD components, attribute to these  programs. While OSD is faced with making funding choices among  competing leadership initiatives, such as its efforts to conduct software  acquisition maturity assessments and collect software metrics, these are  some of the very tasks that are embedded within an effective SPI program.  Thus, by ensuring that DOD components have effective SPI programs, OSD  can leverage programs to indirectly accomplish its other high-priority  initiatives as well.
section_title: Recommendations for Executive Action
paragraphs: To strengthen DLA, Marine Corps, and Navy software and systems  development, acquisition, and engineering processes, we recommend that  the Secretary of Defense direct the Director of DLA, the Commandant of  the Marine Corps, and the Secretary of the Navy to establish SPI programs  where this report shows none currently exist. In so doing, these officials  should consider following the best practices embodied in the SEI IDEALSM  model and drawing from the experiences of the Army, Air Force, DFAS, and  some Navy units.
Further, to strengthen DOD-wide SPI, we recommend that the Secretary of  Defense direct the Assistant Secretary of Defense for Command, Control,  Communications, and Intelligence, in collaboration with the Under  Secretary of Defense for Acquisition, Technology, and Logistics, to (1) issue  a policy requiring DOD components that are responsible for  systems/software development, acquisition, or engineering to implement  SPI programs, and (2) develop and issue SPI guidance and, in doing so,  consider basing this guidance on the SEI IDEALSM model and the positive  examples of SPI within the Army, Air Force, DFAS, and some Navy units  cited in this report.
We also recommend that the Secretary direct the Assistant Secretary for  Command, Control, Communications, and Intelligence to (1) annually  determine the components’ compliance with the SPI policy and  (2) establish and promote a means for sharing SPI lessons learned and best  practices knowledge throughout DOD.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, the Deputy Assistant  Secretary of Defense for Command, Control, Communications, and  Intelligence, who is also the DOD Deputy Chief Information Officer (CIO),  agreed with the report’s message that SPI practices should be used and  encouraged, and that information about SPI practices should be shared  among DOD components. To this end, and since receiving a draft of this  report, the Deputy CIO stated that the Under Secretary of Defense  (Acquisition, Technology, and Logistics) has established a working group  that is, among other things, to develop a plan for implementing SPI.  According to the Deputy CIO, this plan will be ready for internal review in  April 2001.
Further, the Deputy CIO stated that a January 2001 revision to DOD  Regulation 5000.2-R represents a policy step toward addressing software  improvement by including in the regulation a section on software  management. According to the Deputy CIO, while this section does not  specifically call for an SPI program, the regulation provides guidance for  improving software by using, for example, SEI’s Capability Maturity Model  level 3 or its equivalent for major acquisition programs with procurement  costs in excess of $2.19 billion.
In light of the above, the Deputy CIO stated that DOD agreed with our  recommendation to establish and promote a means for sharing SPI lessons  learned and best practices knowledge throughout DOD, and added that a  DOD steering group, which was chartered during the course of our  review, has been assigned responsibility for this function. However, the  Deputy CIO disagreed with our recommendation that DOD issue a policy to  mandate SPI programs for all DOD components and their relevant  activities. According to the Deputy CIO, establishing a policy requiring or  otherwise directing DOD components that do not have SPI programs to  implement them would be premature at this time because there are  insufficient data to justify the sole use of the SEI IDEALSM model and that  unless a specific model were used, compliance with such a policy or  directive would be problematic. Therefore, the Deputy CIO stated a  decision regarding the issuance of DOD-wide policy mandating the  implementation of SPI programs would not be made until the work group  reports its results and develops its plan for implementing SPI. At this point  and without the work group’s findings, according to the Deputy CIO,  issuance of SPI guidance (as opposed to “policy”) would be “a more  beneficial approach.”
In our view, the Deputy CIO’s comments are not inconsistent with our  recommendations, and our point of disagreement appears to center around  simply the timing of actions rather than the recommended actions  themselves. Specifically, while we continue to believe that sufficient bases  currently exist for issuance of a DOD SPI policy requirement, especially in  light of the evidence in our report that (1) without this requirement not all  components are implementing SPI and (2) those components that are  currently implementing SPI are reporting substantial benefits, it is  reasonable for DOD to await its work group’s results before making a  decision on how to proceed. Further, we agree with the Deputy CIO’s  comment that there are insufficient data to justify citing in DOD policy the  SEI IDEALSM model as the single model for SPI. Our report recognizes that  not all of the DOD components that we cited as having effective SPI  programs are using the same model. As a result, our recommendations did  not prescribe a specific SPI model. Instead, we recommended that in  developing SPI policy and associated guidance, DOD should consider  basing this guidance on the SEI IDEALSM model as well as the positive  examples of SPI within the Army, Air Force, DFAS, and some Navy units  cited in the report.
Regarding the Deputy CIO’s comment that DOD has recently revised DOD  Regulation 5000.2-R to include guidance for improving software  management through the use of, for example, SEI’s Capability Maturity  Model level 3, we note that level 3 requirements include performance of  process improvement practices that are expanded upon by the SEI  IDEALSM model. Additionally, we note that the regulation does not apply to  all DOD software/system programs but, rather, only to acquisition  programs that exceed a certain dollar threshold. Therefore, the revised  regulation does not fulfill the intent of our recommendations.
DOD’s written comments, along with our responses, are reproduced in  appendix II.
We are sending copies of this report to Senator John Warner, Senator Carl  Levin, Senator Ted Stevens, Senator Daniel Inouye, and to Representative  Bob Stump, Representative Ike Skelton, and Representative C.W. Bill  Young, in their capacities as Chairmen, Ranking Members, or Ranking  Minority Members of Senate and House Committees and Subcommittees.  In addition, we are sending copies of this report to the Secretaries of the  Army, Navy, and Air Force; the Commandant of the Marine Corps; the  Directors of DLA and DFAS; and the Director, Office of Management and  Budget. Copies will also be available at GAO’s web site, www.gao.gov.
If you have any questions about this report, please contact me at (202) 512- 3439 or by e-mail at hiter@gao.gov. Key contributors to this report are listed  in appendix V.
section_title: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) compare selected DOD components’ SPI  programs against SEI’s IDEALSM model, which is a recognized best  practices model; (2) determine how these components have approached  management of their SPI programs and what program results they are  reporting; and (3) determine what DOD-wide efforts are under way to  promote and leverage the components’ SPI programs. The selected  components include all four services—Army, Air Force, Navy, Marine  Corps—and two DOD agencies that have large, software-intensive system  modernization programs under way—the Defense Finance and Accounting  Service (DFAS) and the Defense Logistics Agency (DLA).
To address the first objective, we reviewed the components’ respective  information technology strategic plans as well as available SPI policies,  guidance, and program documentation, and interviewed headquarters  officials from each component. Using this information, we first ascertained  whether SPI programs or activities existed for a component, and if so, how  they were organized and structured. For the components in which we  found SPI programs or activities, we then identified the units within the  components responsible for implementing those programs and activities.  In instances in which these responsibilities were decentralized (Army, Air  Force, and Navy), we worked with component headquarters and command  officials to select at least two units in each component that collectively  (1) had missions involving both software-intensive weapons and business  systems and (2) were responsible for the largest percentages of software  and systems development, acquisition, and engineering activities within  each component. Table 2 shows the DOD components and  software/systems units where we reviewed SPI programs and activities.  Where “not applicable” is indicated in the table, SPI responsibility resided  at the “Command/major organizational unit,” and therefore our work did  not extend to a “Software/systems unit.”
For each unit that we identified as being responsible for implementing an  SPI program or activities, we analyzed relevant SPI program  documentation, including program descriptions, plans, budgets, and  progress and performance measures and reports, and interviewed program  officials. We then compared this information with the SPI tasks specified  and described in SEI’s IDEALSM model to determine whether the program  satisfied the model.
Designed to assist organizations in implementing and managing effective  SPI programs, the SEI-developed IDEALSM model comprises five specific  phases; a sixth element addresses overall management of the five phases.  Table 3 provides more information about the tasks involved in each phase.  Table 4 lists every task included under each phase.
To address the second objective, we analyzed the aforementioned  information, conducted additional interviews, and reviewed additional  program information from the component units to which SPI management  responsibility had been delegated. As part of this objective, we also  reviewed program progress and performance reports and discussed  program accomplishments with responsible officials to identify examples  of SPI benefits. We then analyzed each component’s SPI program results in  relation to its program management approach to determine whether any  patterns were evident. We did not independently validate components’  reported accomplishments and benefits.
To address the third objective, we interviewed responsible component  officials, reviewed supporting records and documentation, and visited  Internet sites to identify SPI program best practices and lessons learned,  along with what efforts are being made to share these with other activities  and components throughout the department. We also identified two offices  within the Office of the Secretary of Defense (OSD) that have responsibility  and activities underway relating to the advancement of software and  system management practices in the departmentthe Office of the Deputy  Under Secretary of Defense for Acquisition, Technology, and Logistics; and  the Office of the Assistant Secretary of Defense for Command, Control,  Communications, and Intelligence. For each office, we analyzed  documentation describing their respective ongoing and planned activities  and interviewed officials. In doing so, we focused on identifying any  activities that specifically promoted and leveraged SPI programs and  activities under way throughout DOD. We also discussed with SPI program  officials in each component their awareness of the OSD efforts.
We performed our work at Army headquarters, the Pentagon, Arlington,  Virginia; and interviewed officials and reviewed documentation from the  Communications-Electronics Command Software Engineering Center at  Fort Monmouth, New Jersey; and the Aviation and Missile Command  Software Engineering Directorate at Redstone Arsenal, Alabama. We also  performed our work at Navy headquarters in Arlington, Virginia; and  interviewed officials and reviewed documentation from the Naval Aviation  Systems Command at Patuxent River, Maryland; and the Space and Naval  Warfare Systems Command Centers at San Diego, California; Chesapeake,  Virginia; and Charleston, South Carolina. We also interviewed officials and  reviewed documentation from the Air Force’s Electronic Systems Center  Standard Systems Group at Maxwell Air Force Base, Alabama; the Materiel  Systems Group at Wright-Patterson Air Force Base, Ohio; and the Air Force  Academy in Colorado Springs, Colorado. We also performed our work at  Marine Corps headquarters in Arlington, Virginia; and interviewed officials  and reviewed documentation from the Marine Corps Systems Command in  Quantico, Virginia; and the Marine Corps Tactical Systems Support Activity  at Camp Pendleton, California. We also performed work at DFAS  headquarters in Arlington, Virginia; and DLA headquarters at Fort Belvoir,  Virginia. We conducted our work from March through December 2000, in  accordance with generally accepted government auditing standards.
section_title: Comments From the Department of Defense
paragraphs: The following are GAO’s comments on the Department of Defense’s letter  dated March 2, 2001.
subsections: section_title: GAO Comments
paragraphs: 1. We disagree. Sufficient bases currently exist for issuance of a DOD SPI  policy requirement, especially in light of the evidence in our report that  (1) without this requirement not all components are implementing SPI  and (2) those components that are currently implementing SPI are  reporting substantial benefits. Nevertheless, DOD's decision to await  an OSD work group's results before making a decision on how to  proceed is not unreasonable or inconsistent with our position.  2. See response to comment 1. 3. We disagree. Oversight is an important part of policy implementation,  and without such oversight, DOD would incur significant risk that the  policy would not be implemented. Further, establishing a baseline  measure to determine compliance does not require the implementation  of a specific model. The intent of our recommendations is to establish a  policy requiring SPI that recognizes, as our report recognizes, that there  is more than one model for doing so effectively.
section_title: Description of SEI Capability Maturity Models
paragraphs: Since 1984, the Software Engineering Institute (SEI) has worked to  improve management of software/systems productivity and quality  primarily by addressing problems in acquiring, developing, engineering, or  enhancing software/systems through a series of capability maturity models.  According to SEI, an organization’s process capability provides a means of  predicting the most likely outcome of the next software/systems project  undertaken; process maturity implies that the productivity and quality  resulting from an organization’s software/systems processes can be  improved as maturity of the processes increases. The IDEALSM model is  based on lessons learned from SEI experiences as well as from SEI projects  relating to software process capability and maturity. For example, during  the initiating phase of the IDEALSM model, general SPI program goals are  defined, and this definition could be in terms of capability maturity model  levels. In the diagnosing phase, IDEALSM recommends developing an  organization process maturity baseline; SEI’s capability maturity model− based appraisal is one way of establishing this baseline.
The first of these capability maturity models, the Software Capability  Maturity Model® (SW-CMM®), was designed to assist organizations in  improving software development and maintenance processes. In this  model, software process maturity—ranked from a low of level 1 to a high of  level 5—serves as an indicator of the likely range of software cost,  schedule, and quality that can be expected to be achieved by projects  developed within an organization. (See figure 2.)
Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.
subsections: section_title: Predictable process
paragraphs: Detailed measures of the software process and product quality are collected.  Both the software process and products are quantitatively understood and controlled.
section_title: Standard, consistent process
paragraphs: The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization.  All projects use an approved, tailored version of the organization’s standard software process for developing and maintaining software.
section_title: Disciplined process
paragraphs: Basic project management processes are established to track cost, schedule, and functionality.  The necessary process discipline is in place to repeat earlier successes on projects with similar applications.
The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort.
Since the SW-CMM® was published, SEI has developed additional models  in the capability maturity series:   The Software Acquisition CMM® is a model for improving the software  acquisition process. It follows the same five-level architecture as the  SW-CMM® but emphasizes acquisition issues and the needs of  individuals and groups planning and managing software acquisition  activities.
The Systems Engineering CMM® describes the essential elements of an  organization’s systems engineering process and provides a reference for  comparing actual systems engineering practices against these elements.  The model addresses the process aspects of systems engineering and  the product development portion of the life cycle. This model was a  collaboration of several organizations, including SEI.
In 1997 a team led by DOD, in conjunction with SEI, government, and  industry, concentrated on developing an integrated framework for  maturity models and associated products. The result was the CMM  IntegrationSM (CMMISM), which is intended to provide guidance for  improving an organization’s processes and the ability to manage the  development, acquisition, and maintenance of products and services,  while reducing the redundancy and inconsistency caused by using  stand-alone models.
The CMMISM combines earlier models from SEI and the Electronic  Industries Alliance into a single model for use by organizations pursuing  enterprise-wide process improvement. However, the prototype CMMISM  does not include the acquisition features of the SA-CMM® because the  team wanted to focus first on the development process. A CMMISM that  includes coverage for acquiring software-intensive systems is currently  being developed. Additional disciplines may also be covered. Ultimately,  the CMMISM is to replace the models that have been its starting point.
section_title: Detailed Results of Review of DOD Components’ SPI Programs
subsections: section_title: Army SPI Program
subsections: section_title: Background
paragraphs: The Army depends on software-intensive systems to support each of its  major commands and every facet of its mission, from weapons to financial  management systems. The Army budgeted about $3.3 billion on  information technology during fiscal year 2000.
The Army has assigned responsibility for information systems to the Army  Materiel Command (AMC). Several major subcommands function under  AMC. Three of these major subcommands—the Communications- Electronics Command (CECOM), the Aviation and Missile Command  (AMCOM), and the Tank-Automotive Command—are responsible for the  acquisition, development, engineering, and maintenance of information  technology for the Army. We reviewed the Army’s SPI activities at CECOM  and AMCOM.
CECOM has assigned responsibility for information systems to its Software  Engineering Center (SEC). The center, located at Fort Monmouth, New  Jersey, is supported by several software/systems activities located across  the United States. The center is responsible for overseeing about 85  percent of the Army’s systems, including (1) command, control,  communications, and computers; (2) intelligence, electronic warfare, and  sensors; (3) sustaining base/power projection; and (4) AMC business  systems.
AMCOM has assigned responsibility for its information systems to its  Software Engineering Directorate (SED). This directorate, located at  Redstone Arsenal, Alabama, oversees and provides life-cycle support to  both aviation and missile weapons systems. (See figure 3.)
Army’s SPI program activities began in the early 1990s; in mid-1996 the  Secretary mandated that all Army software acquisition, development, and  maintenance activities establish SPI programs. At the same time, the Army  published an SPI policy that specified two requirements:  First, a contractor’s capability to produce quality software will be part of  the Army’s source-selection evaluation process. The Army has  implemented this requirement by evaluating potential contractors  against SW-CMM® level 3 maturity requirements and requiring  contractors that do not meet these requirements to propose a strategy  for mitigating the risks associated with not meeting them. This  requirement is further enforced during milestone reviews of major  systems, when the program manager must show that the contractor  meets these requirements.
Second, Army software activities will continually improve their software  and systems process maturity, including self-assessments of existing  processes, and achieve SW-CMM® level 3 within 6 years of initial  assessment.
section_title: Army’s SPI Program Is Aligned With SEI’s IDEALSM Model
paragraphs: Both the CECOM SEC and the AMCOM SED SPI programs are fully  consistent with the IDEALSM model. Table 5 shows examples of program  elements that reflect some of the recommended tasks in the IDEALSM  model; table 6 provides a detailed comparison of CECOM and AMCOM’s  SPI programs against each of the IDEALSM model recommended tasks.
section_title: Army Reports That Its Decentralized Approach to SPI Program Management Has Produced Results
paragraphs: When the Army first launched its SPI activities, it managed initiation and  diagnosis centrally, with both CECOM and AMCOM being involved in these  early actions. Subsequently, as many groups throughout the Army were  trained in using the SEI process maturity measurements, responsibility for  implementing SPI programs was delegated to the commands. The Army has  since expanded this decentralized approach, giving each command the SPI  requirements through Army policy and allowing each to implement the  policy as it determines best supports its mission.
According to information that these two subcommands provided, their SPI  programs have produced positive results. One of AMCOM’s measures of  software quality is development productivity, which is the number of lines  of software code produced as a function of resources invested. According  to AMCOM, SED’s productivity ratio for new development products  increased from 1.30 to 2.48 as a result of moving from SW-CMM® level 2 to  level 3. SED reports that it has recently achieved level 4.
section_title: Air Force SPI Program
subsections: section_title: Background
paragraphs: Software-intensive systems are vital to the Air Force’s overall mission.  They are used to sustain weapons systems, airborne electronics, electronic  warfare, space communications, and support equipment. The Air Force has  about 1,600 systems and budgeted about $4.6 billion in fiscal year 2000 for  information technology.
The Air Force has nine major commands, but its largest software/systems  units are under the Air Force Materiel Command (AFMC). Within AFMC,  we reviewed SPI efforts at two units within the Electronic Systems Center,  which provides command and control and information systems for the Air  Force as well as for other DOD units, using a budget of over $3 billion in  fiscal year 2000. The two units that we reviewed were the Standard  Systems Group (SSG) at Montgomery, Alabama, and the Materiel Systems  Group (MSG) at Dayton, Ohio. In addition, we reviewed SPI activities at the  Air Force Academy (AFA), which has one of the remaining  software/systems units outside AFMC. (See figure 4.)
SSG is the largest software/systems unit within the Air Force in terms of  money invested and amount of software delivered. Its mission is to develop  and maintain combat support information systems for the Air Force and  other DOD components. Additionally, SSG manages information  technology contracts and standard information systems programs  commonly used at all active and reserve Air Force bases and some DOD  agencies worldwide.
Next to SSG, MSG is the largest Air Force central software/systems unit.  MSG’s mission is to support the Air Force goal of information dominance  through acquiring, developing, maintaining, reengineering, and providing  technical services for information systems.
AFA has a software/systems unit that is primarily responsible for  maintaining and developing the Cadet Administrative Management  Information System, a mission-critical database system that tracks the  progress of cadets from precandidacy through academic, physical,  ethical/moral, and military training programs and, after graduation,  throughout their Air Force careers.
In 1991, the Deputy Assistant Secretary of the Air Force initiated the  service’s SPI program. In particular, Air Force software/systems units were  directed to complete SW-CMM® assessments by October 1, 1994, perform  follow-up assessments every 2 years, and achieve SW-CMM® level 3 by  1998. Air Force’s 1994 SPI policy was revised this year. This revised policy  requires all units that develop or maintain software/systems to have an SPI  program and a documented SPI plan that includes, at least, a baseline  measure of their current capabilities, goals and milestones they intend to  reach, and metrics with which to measure their progress toward goals and  milestones.
section_title: Air Force SPI Program Is Aligned With SEI’s IDEALSM Model
paragraphs: The IDEALSM model is the framework the Air Force recommends to its  software/systems units, and our comparison of the activities at SSG, MSG,  and AFA to the IDEALSM model found that their respective SPI programs  are almost all aligned with the model. Specifically, each of the programs  satisfied all but five of the IDEALSM model recommended tasks, and none of  those five is significant enough to preclude having effective SPI programs.  Table 7 shows examples of the programs’ elements that reflect some of the  recommended tasks in the IDEALSM model; table 8 shows a detailed  comparison of SSG, MSG, and AFA SPI programs against each of the  IDEALSM model recommended tasks.
section_title: Air Force Reports That Its Decentralized Approach to SPI Program Management Has Produced Results
paragraphs: Air Force headquarters has delegated SPI responsibility to its  software/systems units. When the Air Force began its SPI activities in 1991,  its goal was to initiate SPI by performing assessments that would indicate  the current level of maturity at Air Force units. Management of this effort  was centralized in the Air Force Communications Agency (AFCA). AFCA  staff visited all 41 Air Force units, some more than once, to perform  assessments. Once software/systems units became capable of conducting  their own process maturity measurements, Air Force began decentralizing  management of the SPI program to the units. The last year in which the Air  Force exercised any centralized management of SPI was 1998.
The Air Force’s SPI efforts have been, in its view, beneficial. For example,  one Air Force center reported a 7.5-to-1 return on its SPI investment, which  was independently verified. An official at another center stated that SPI  had allowed its organization to achieve higher process maturity levels and  made significant improvements in the quality of its software products and  its productivity measures.
section_title: Navy SPI Program
subsections: section_title: Background
paragraphs: The Navy depends on software-intensive systems to support many  functions throughout its nine operating forces—including the Marine  Corps—and its 15 support units—including four major systems commands.  These systems support some aspect of every operation, including strategic  and tactical operations; sophisticated weaponry; intelligence, surveillance,  and security; strategic sealift and fleet mobilization and readiness; and  routine business functions such as finance, personnel, logistics, and  contract management. In fiscal year 2000, the Navy budgeted about  $3.1 billion for information technology.
Within the Navy, acquisition, development, and maintenance of these  systems is delegated to its major systems commands: the Naval Aviation  Systems Command (NAVAIR), Space and Naval Warfare Systems Command  (SPAWAR), Naval Sea Systems Command, and Naval Supply Systems  Command. We reviewed SPI activities at NAVAIR and SPAWAR. Both  commands have several subordinate units involved in acquiring,  developing, and maintaining systems. (See figure 5.) and Acquisition)
NAVAIR provides full life-cycle support to 148 programs, such as aircraft,  avionics, air-launched weapons, electronic warfare, cruise missiles, and  unmanned aerial vehicles. NAVAIR has two divisions (weapons and  aircraft). The weapons division has two California product centers, and the  aircraft division has three centers, located in New Jersey, Maryland, and  Florida.
SPAWAR develops, acquires, and maintains systems through three SPAWAR  Systems Centers (SSC). These centers are at San Diego, California;  Chesapeake, Virginia; and Charleston, South Carolina. We reviewed  SPAWAR’s SPI efforts at all three centers. SSC San Diego develops,  acquires, and supports command, control, communications, and ocean  surveillance systems. SSC Chesapeake develops, acquires, and supports  supply, inventory, finance, food service, and other information systems.  SSC Charleston develops, acquires, and supports command, control,  communications, intelligence, surveillance, and reconnaissance systems.
To guide and direct their respective SPI programs, these commands follow  DOD and other models and standards. Commands have also established  local policy. For instance, SPAWAR policy requires all managers with  software-related responsibilities at San Diego to incorporate process  improvement in the areas of new development, modification, reuse,  reengineering, maintenance, integration, and all other activities resulting in  software products. In 2000, NAVAIR published an interim policy that  requires prospective contractors to be evaluated at SEI SW-CMM® level 3  for all acquisitions.
section_title: Navy’s SPI Program Is Partly Aligned With the IDEALSM Model
paragraphs: Navy’s experience with SPI to date has been mixed. Both SSC San Diego  and NAVAIR have SPI programs that are consistent with the IDEALSM  model. However, SSC Chesapeake’s and SSC Charleston’s programs are  not. Specifically, SSC Chesapeake has only recently initiated an SPI  program and, while efforts to date are aligned with the IDEALSM model,  many important SPI program tasks have yet to be executed. For example,  in July 2000 it completed some initiating-phase tasks, such as creating a  management steering group and an SEPG. However, it has yet, for example,  to (1) conduct baselines to identify process strengths and weaknesses in  the diagnosing phase, (2) develop an SPI plan with measurable goals and  committed resources in the establishing phase, (3) pilot-test potential  solutions or transition the solutions to long-term support in the acting  phase, or (4) gather or analyze lessons learned in the leveraging phase.
In the case of SSC Charleston, no SPI program exists, although the center  has undertaken one task that is intended to begin the initiating phase of a  program. Table 9 shows examples of Navy SPI programs’ elements that  reflect some of the recommended tasks in the IDEALSM model; table 10  shows a detailed comparison of NAVAIR and SPAWAR SPI programs  against each of the IDEALSM model recommended tasks.
section_title: Navy Reports That Its Decentralized Approach to SPI Program Management Has Produced Results
paragraphs: The Navy has delegated SPI responsibility to its commands, which in some  cases have further decentralized SPI program management within the  command structure. For example, NAVAIR manages its SPI program  centrally through its Software Process Improvement Office. Established in  1999, this office, in combination with two NAVAIR executive groups,  establishes NAVAIR software improvement policies, monitors  performance, and provides support for process training and baselining. In  contrast to NAVAIR, SPAWAR decentralized SPI program management to  its SSCs.
Navy reports several SPI program benefits. For example, officials at  NAVAIR’s F/A-18 software program report reaching SW-CMM® level 3 with  benefits including cost savings, improved product quality, and a 7:1 return  on their SPI investment. In addition, SSC San Diego officials report that  their SPI program significantly reduced both the number of software  defects and the time expended in testing their air traffic control program  system. In particular, staff months spent addressing trouble reports were  reduced by 70 percent. These officials also state that benefits from SPI  include better management control, improved overall software  performance, higher customer satisfaction, and increased competitive  advantage and repeat business.
section_title: Marine Corps SPI Program
subsections: section_title: Background
paragraphs: The Marine Corps depends on software-intensive systems to support every  facet of its mission—from weapons to tactical communications systems. In  fiscal year 2000, the Marine Corps budgeted about $525 million for  information technology.
The Marine Corps has assigned responsibility for acquisition, development,  engineering, and maintenance of information technology to the Marine  Corps Systems Command. The Command is the sole procurement activity  for the Marine Corps, purchasing everything from business systems to  software-intensive weaponry such as tanks and command, control,  communications, and computer equipment. The Marine Corps Tactical  Systems Support Activity (MCTSSA), located at Camp Pendleton,  California, is a subordinate command. MCTSSA is responsible for software  life-cycle support of designated Marine Corps and joint-service tactical data  systems and software. (See figure 6.)
According to MCTSSA officials, the Marine Corps does not have a formal  SPI program, although it has performed SPI activities since the early 1990s.  MCTSSA uses both DOD and Marine Corps guidance to manage its SPI  activities. At one time, however, MCTSSA appeared to be on its way to a  formal SPI program. It started SPI activities in the early 1990s, and by 1995  was using SEI to support them. For example, during 1995 and 1996 SEI  assisted the Marine Corps in identifying program weaknesses and in  developing solutions to improve them. However, MCTSSA officials stated  that they did not renew the SEI contract because of a lack of funds.
section_title: Marine Corps SPI Activities Are Partly Aligned With SEI’s IDEALSM Model
paragraphs: MCTSSA is performing many of the tasks recommended by the IDEALSM  model. Table 11 shows examples of activities that reflect some of the  recommended tasks in the IDEALSM model; however, in all but the  diagnosing phase, MCTSSA is not executing some key recommended tasks.  For example, (1) in the initiating phase, it has not defined general SPI goals  or guiding principles; (2) in the establishing phase, it has not developed an  SPI plan with measurable goals and committed resources; (3) in the acting  phase, it developed solutions but never pilot-tested potential solutions or  transitioned the solutions to long-term support; and (4) in the leveraging  phase, it gathered lessons learned but did not analyze them or use them to  revise its organizational approach. Further, it has not reviewed its  sponsorship and commitment, established high-level goals, or decided to  continue with the SPI process. Without performing these steps, it is  unlikely that SPI activities will produce the kind of meaningful advances in  product quality and cost savings that other DOD components have realized.  Table 12 shows a detailed comparison of MCTSSA SPI activities against  each of the IDEALSM model recommended tasks.
section_title: Marine Corps Uses a Decentralized Approach to Manage SPI Program Activities
paragraphs: The Marine Corps has adopted a decentralized management approach for  SPI by delegating responsibility to the Command, which in turn has  delegated most of the responsibility to MCTSSA. Specifically, the  Command retained overall responsibility for SPI but assigned other  activities, such as defining standard processes or metrics, to MCTSSA.
section_title: DFAS’ SPI Program
subsections: section_title: Background
paragraphs: DFAS provides finance and accounting services to DOD components.  Created in 1991, it replaced more than 300 service and agency finance and  accounting offices that operated more than 300 systems. DFAS consists of  five centers and 20 field offices; it operates 83 finance and accounting  systems but plans to reduce this number to about 30 or fewer by the end of  2005. The systems are acquired and maintained by seven software/systems  units called systems engineering organizations (SEOs). (See figure 7.) In  fiscal year 2000, DFAS budgeted about $225 million for information  services.
DFAS began its SPI program in 1993 when responsibility for SPI was  assigned to the Financial Systems Organization (FSO) and a corporate  SEPG was established to manage the program and coordinate SPI among  FSO field locations. FSO established an SPI policy in its 1995 SPI strategic  action plan. That policy is currently under revision. The latest draft  revision has continuous SPI as an objective for all DFAS SEOs. The policy  also requires that best practices be shared across all DFAS SEOs and that  process metrics be collected, maintained, analyzed, used, and reported to  support the SPI process.
section_title: DFAS SPI Program Is Aligned With SEI’s IDEALSM Model
paragraphs: DFAS’ SPI program is fully consistent with the IDEALSM model. Table 13  shows examples of DFAS activities that reflect some of the recommended  IDEALSM tasks; table 14 shows a detailed comparison of DFAS SPI activities  with each of the IDEALSM model recommended tasks.
section_title: DFAS Reports That Its Centralized Approach to SPI Program Management Has Produced Results
paragraphs: DFAS centralized SPI program management when it started its program  under FSO in 1993. When reorganized in 1998, DFAS retained centralized  SPI management by assigning it to the Infrastructure Services Organization  (ISO). Under the draft SPI policy revision, management of the program will  still be centralized in headquarters, but responsibilities will be split  between ISO and its parent unit, the Information and Technology  Directorate (ITD). Specifically, the draft revision assigns the ITD  responsibility for approving SPI policies, maintaining metrics, and  analyzing metrics for the purpose of recommending changes in priorities,  resources, and processes. The draft revision assigns the ISO responsibility  for publishing SPI policies, maintaining the agency process assets library,  and coordinating DFAS-wide SPI activities.
ITD reports that its SPI program has improved DFAS staff productivity. A  DFAS contractor has conducted benchmarking measurements on DFAS  software/systems development efforts. Results from 1996 and 1997  measurements show that DFAS develops and maintains software  (measured in terms of function points) for $0.67 that costs an average  organization $1.00, a comparable government organization $1.06, and a  comparable large commercial organization $1.37. The contractor cited  “strong processes” as one factor that contributed to DFAS’ productivity.
section_title: DLA’s SPI Program
subsections: section_title: Background
paragraphs: DLA is a combat support agency whose primary role is to provide supply  management, logistics services, and distribution support to America’s  military forces worldwide. It relies on software-intensive systems to  administer over $900 billion in DOD and other agency contracts. DLA  budgeted about $784 million for information technology in fiscal year 2000.
In 1998, DLA’s systems design center operated nine systems development  and maintenance units across the country. After closing this center in  December 1998 in order to streamline operations and reduce costs, DLA  created three systems integration offices to oversee the development and  maintenance units, which have been cut from nine to seven. (See figure 8.)  Each of the three offices supports software development and maintenance  for a separate DLA business function—materiel management, logistics, and  base support and distribution.
DLA does not have an SPI program, having eliminated the program in 1998  when its system design center was closed in 1998. However, as part of its  ongoing reorganization, DLA rewrote the policy and duties of its CIO and  moved that function to the new Information Operations unit. The CIO told  us that the SPI program is to be reestablished. However, specific plans and  milestones for doing so were not available.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  were Tonia Brown, Suzanne Burns, Michael Fruitman, John Ortiz, Madhav  Panwar, and Teresa Tucker.
section_title: Ordering Information
paragraphs: The first copy of each GAO report is free. Additional copies of  reports are $2 each. A check or money order should be made out to  the Superintendent of Documents. VISA and MasterCard credit  cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are  discounted 25 percent.
Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders by phone: (202) 512-6000 fax: (202) 512-6061 TDD (202) 512-2537 Each day, GAO issues a list of newly available reports and  testimony. To receive facsimile copies of the daily list or any list  from the past 30 days, please call (202) 512-6000 using a touchtone  phone. A recorded menu will provide information on how to obtain  these lists.
section_title: To Report Fraud, Waste, or Abuse in Federal Programs
paragraphs: Web site: http://www.gao.gov/fraudnet/fraudnet.htm  e-mail: fraudnet@gao.gov  1-800-424-5454 (automated answering system)