DOD Systems Modernization: Uncertain Joint Use and Marginal Expected Value of Military Asset Deployment System Warrant Reassessment of Planned Investment
Published: 2005-12-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Because of the importance of the Department of Defense's (DOD) adherence to disciplined information technology (IT) acquisition processes in successfully modernizing its business systems, GAO was asked to determine whether the Transportation Coordinators' Automated Information for Movements System II (TC-AIMS II) program is being managed according to important aspects of DOD's acquisition policies and guidance, as well as other relevant acquisition management best practices. TC-AIMS II was initiated in 1995 as a joint services system to help manage force and equipment movements within the United States and abroad. The U.S. Department of the Army has the lead responsibility for managing the system's acquisition and estimates its life-cycle cost to be $1.7 billion over 25 years.
section_title: What GAO Found
paragraphs: The Army has managed the TC-AIMS II program in accordance with some, but not all, key aspects of DOD's system acquisition management policies and related guidance. These policies and guidance are intended to reasonably ensure that investment in a given IT system represents the right solution to fill a mission need--and, if it does, that acquisition and deployment of the system are handled in a manner that maximizes the chances of delivering defined system capabilities on time and within budget. The Army has not managed the program in accordance with those DOD policies and related guidance, including related federal and other best practice guidance, that are intended to reasonably ensure that a proposed system is the right solution to meet mission needs. Specifically, the Army has not economically justified its investment in TC-AIMS II on the basis of reliable estimates of costs and benefits. For example, the most recent economic justification included cost and benefit estimates predicated on all four military services using the system. However, two services (U.S. Department of the Air Force and U.S. Marine Corps) have stated that they do not intend to use it. The Army has not invested in TC-AIMS II within the context of a well-defined enterprise architecture, which is an institutional blueprint to control program investment decisions in a way that promotes interoperability and reduces redundancy among systems. The Army has instead focused on aligning TC-AIMS II with its logistics architecture; this means that even though TC-AIMS II is intended to be a DOD-wide program, it has been based on a service-specific architecture rather than a DOD-wide architecture. As a result, it may not properly fit within departmentwide plans. To its credit, the Army has largely managed the program in accordance with key policies and related guidance that are intended to reasonably ensure that the acquisition and deployment of a given system are handled in a manner that maximizes the chances of delivering defined capabilities on time and within budget. However, some aspects of this policy and guidance have not been followed. For example, the Army has not fully implemented risk management and has not adhered to a key feature of performance-based contracting. Reasons the Army cited for not following policies and guidance ranged from management inattention to lack of training. As a result, the Army, among other things, does not know whether the system is the right solution. Until this uncertainty and the previously discussed problems are addressed, it will remain unclear whether further planned investment in TC-AIMS II is warranted, and certain aspects of the program's management will be limited.

section_title: Letter
paragraphs: Because it is so important that the Department of Defense (DOD) adhere to  disciplined information technology (IT) acquisition processes in order to  successfully modernize its business systems, you requested that we  determine whether the department is following its own IT acquisition  policies and guidance, which were  recently revised. As part of our  response to your request, we agreed to review the Transportation  Coordinators’ Automated Information for Movements System II (TC-AIMS  II) program. TC-AIMS II was initiated in 1995 by DOD as a joint services  system with the goal of helping to manage the movement of forces and  equipment within the United States and abroad. Assigned the lead  responsibility for acquiring the system, the U.S. Department of the Army  estimates the system has a 25-year, life-cycle cost of $1.7 billion.
Our objective was to determine whether TC-AIMS II is being managed  according to important aspects of DOD’s acquisition policies and guidance  as well as other relevant acquisition management best practices. We  focused on the program’s (1) economic justification; (2) architectural  alignment; (3) risk management; (4) requirements development and  management; (5) commercial component management; and (6) contract  management, including contractor oversight and performance-based  contracting.
We conducted our work between October 2004 and October 2005, in  accordance with generally accepted government auditing standards.  Details of our objective, scope, and methodology are included in  appendix I.
subsections: section_title: Background
paragraphs: As one of the largest and most complex organizations in the world, DOD  spends billions of dollars each year to operate, maintain, and modernize IT  systems that support core business operations. These business operations  consist of various interrelated and interdependent business functions,  including logistics management, procurement, health care management,  and financial management. Further, execution of these business operations  span a wide range of defense organizations, including the military services  and their respective major commands and functional activities, defense  agencies and field activities, and combatant and joint operational  commands that are responsible for military operations in geographic  regions or theaters. For fiscal year 2005, DOD requested approximately $13  billion to operate, maintain, and modernize about 4,150 business systems  and related IT infrastructure. The Army’s share of this funding and systems  is about $2.7 billion and 727 systems.
In 1995, we designated DOD’s business systems modernization efforts as a  high-risk program, and we continue to designate it as such today for  several reasons, including the department’s challenges in implementing  effective IT investment management structures and processes, developing  and implementing an enterprise architecture, and implementing effective  IT system acquisition and development processes.
subsections: section_title: Overview of TC-AIMS II Genesis, Purpose, and Acquisition Strategy
paragraphs: In the early 1990s, DOD recognized the need to integrate defense IT  systems by ordering the consolidation of systems that serve multiservice  purposes. Accordingly, it directed that approximately 150 operating  transportation systems be evaluated for consolidation opportunities. In  1995, DOD initiated TC-AIMS II as one of a number of programs that were  commenced following this evaluation to consolidate and replace existing  systems, and to automate transportation management function areas for all  services. In doing so, DOD designated the Army as the “executive agent”  responsible for managing TC-AIMS II, including acquiring the system.
Expected TC-AIMS II benefits included enhancing and improving the  efficiency and effectiveness of the support planning needed to deploy and  redeploy forces and equipment, both within the United States and abroad;  improving the visibility of assets; and enhancing cargo and passenger  receiving, controlling, and shipping. The system is also expected to enable  decision makers at various command levels to access information about  unit movement and installation transportation, and is to provide key shared  functionality related, for example, to load planning.
The Army defined the following five-block acquisition and implementation  strategy for TC-AIMS II. According to the strategy, each block is to address  a functional portion of the requirements.
Block 1: This block is to import data about assets (people, equipment,  and supplies) from the services’ existing systems and  is to manage  these data to (1) plan and execute organizational moves (i.e., provide  documentation, packing and shipping labels, and orders) and (2) request  and schedule strategic and local transportation vehicles (e.g., trucks,  ships, airplanes, motor pools, and trucks).
Block 2: This block is to add Web access to permit automated data  exchange with related systems, such as airlift load planning systems and  global transportation networks. This increment is to link and track  cargo and personnel (via DOD’s standard smart cards), and their  corresponding transports. It also is to generate and manage the  documentation associated with these functions.
Block 3: This block is to add interfaces to both supply and personnel  systems to facilitate control over “in theater” operations, such as  reception (unit arrival and off-loading in theater), staging (preparation  for departure to assigned locations), onward movement (transit to  assigned locations), and integration (coordinating and linking the  movement of multiple related units). It also is to generate and manage  documentation associated with these functions.
Block 4: This block is to add the capability to manage maritime forces  associated with prepositioned equipment and cargo (e.g., tank  ammunition for an armored cavalry unit), including the ability to  manage cargo logistics (loading and off-loading), as well as greater  capability to manage theater operations, including the availability,  distribution, and dispatch of vehicle fleets and drivers. It also is to  generate and manage documentation associated with these functions.
Block 5: This block is to automate the management of transportation  assets and traffic operations at military installations both in the United  States and worldwide. At this point, TC-AIMS II is to be able to generate  and manage all data and documentation related to moving units within  DOD’s transportation system. As part of this block, the interfaces to  systems that are to be integrated with TC-AIMS II are to be upgraded or  created, as required, in response to changes in technology.
Figure 1 shows the Army’s timeline for acquiring the five TC-AIMS II  blocks. In particular, the Army reports that it is currently acquiring Block 3  with the goal of making it operational by late 2006.
The Army estimates the total life-cycle cost of TC-AIMS II to be $1.7 billion  over 25 years, including $569 million for acquisition and $1,168 million for  operations and maintenance. The Army reports that it has spent  approximately $751 million on TC-AIMS II since its inception.
section_title: Description of TC-AIMS II Roles and Responsibilities
paragraphs: A number of DOD and Army organizations are involved in managing the  TC-AIMS II program and acquisition. Briefly, they are as follows:  The Assistant Secretary of Defense for Network Information and  Integration, who has overall responsibility for the program.
The Army’s Program Executive Office for Enterprise Information  Systems, Transportation Information Systems, oversees the  management of TC-AIMS II, which is managed on a day-to-day basis by  the Project Manager, Joint Program Management Office.
The Joint Requirements Board represents TC-AIMS II users and includes  stakeholders from the military services and other DOD organizations.  The board is responsible for managing TC-AIMS II requirements  development activities.
These and other key organizations and their roles and responsibilities are  described in more detail in table 1.
The Army uses a system integration contractor to support the program  office in managing the acquisition of TC-AIMS II. Most recently, it awarded  a firm, fixed-price contract to a contractor in April 2004 to deliver the Block  3 solution. The contract has 5 option years, which are 1 year in duration. To  help it oversee contractor activities, the program office has employed the  help of the General Service Administration’s Federal Systems Integration  and Management Center, whose mission is to provide acquisition  management expertise and support to DOD and other federal agencies on  large, complex IT projects.
section_title: Prior Review Identified Strengths and Weaknesses in DOD’s Acquisition Policies and Guidance
paragraphs: In July 2004, we reported that DOD’s revised systems acquisition policies  and guidance incorporated many best practices for acquiring business  systems, such as (1) justifying system investments economically, on the  basis of costs, benefits, and risks, and (2) continually measuring an  acquisition’s performance, cost, and schedule against approved baselines.  However, the revised policies and guidance did not incorporate a number  of other best practices, particularly those associated with acquiring  commercial component-based business systems, and DOD did not have  documented plans for incorporating these additional best practices into its  policies. We also reported that the department’s revised acquisition policies  did not include sufficient controls to ensure that military services and  defense agencies would appropriately follow these practices. We  concluded that until these additional best practices were incorporated into  DOD’s acquisition policies and guidance, there was increased risk that  department system acquisitions would not deliver planned capabilities and  benefits on time and within budget, and increased risk that DOD  organizations would not adopt and use best practices that were defined.  Accordingly, we made 14 recommendations to the Secretary of Defense  that were aimed at strengthening DOD’s acquisition policy and guidance by  including additional IT systems acquisition best practices and controls for  ensuring that these best practices were followed. DOD agreed with most of  our recommendations and has since issued additional system acquisition  guidance.
section_title: TC-AIMS II Has Not Been Managed in Accordance with Certain DOD System Acquisition Policies and Related Guidance
paragraphs: The Army, as DOD’s acquisition agent for TC-AIMS II, has not managed this  program in accordance with certain department acquisition policies, and  related guidance, that are intended to reasonably ensure that a proposed  system is the right solution to meet mission needs. In particular, the Army  has not economically justified the program on the basis of reliable  estimates of life-cycle costs and benefits, and it has not ensured that this  program, which is intended to produce a departmentwide military  deployment management system, is aligned with a departmentwide  business enterprise architecture or an integrated blueprint for business  systems modernization. As a result, the Army does not know that  investment in TC-AIMS II as planned is warranted, represents a prudent use  of limited DOD resources, and will be interoperable with and not be  duplicative of related systems.
Even if TC-AIMS II happens to be the right system, both economically and  architecturally, the Army has not fully implemented risk management and  has not fully employed performance-based contracting practices. However,  it has largely managed system requirements, commercial components, and  contractor tracking and oversight in accordance with DOD policies and  related guidance. If the program can be economically justified and  architecturally defined, then fully addressing these key acquisition  management areas will be essential to the program’s success.
subsections: section_title: Investment in TC-AIMS II Has Not Been Adequately Justified on the Basis of Costs and Benefits
paragraphs: DOD’s acquisition management policy and related guidance recognize the  importance of developing reliable economic analyses to support  investment decision making. Accordingly, they describe how these  analyses should be developed and used. For TC-AIMS II, the Army has not  adhered to important aspects of departmental policy and related guidance,  such as updating analyses to reflect material changes in program scope and  justifying investment in large, multiyear programs on an incremental basis.  As a result, the Army currently lacks an adequate basis to justify its planned  investment in TC-AIMS II.
subsections: section_title: Latest Economic Analysis Does Not Recognize System’s Uncertain Joint Use and Shows Marginal Expected Value
paragraphs: DOD policy and guidance state that projects should be economically  justified on the basis of reliable analyses of expected life-cycle costs and  benefits and that cost and benefit estimates should be based on realistic  plans for the program’s scope. Federal guidance also advocates  economically justifying proposed investments on the basis of a benefit-to- cost ratio that is greater than 1, basing that ratio on reliable analyses of  expected, quantifiable life-cycle benefits, costs, and risks. The guidance  also promotes the calculation and consideration of qualitative benefits in  preparing an economic analysis. Further, the guidance calls for updating  the analysis when significant program changes occur. These practices help  to continually ensure a positive return on investment—even as program  changes occur.
The program office developed economic analyses to justify investment in  the system in August 2002 and again in December 2003. The program office  was supported in its efforts by the Office of the Deputy Assistant Secretary  of the Army’s Office of Cost and Economics and the Office of the Secretary  of Defense’s Office of Program Analysis and Evaluation, whose missions  include verifying and validating the reliability of cost and benefit estimates  found in Army and other services’ economic analyses. However, this most  recent economic analysis is not based on reliable estimates of life-cycle  costs and benefits. Specifically, the analysis was based on the assumption  that TC-AIMS II would be used by all four military services, and, thus, it  included life-cycle cost estimates showing all of the services’ data—as well  as the benefits that would accrue through its use by all four. However, this  assumption is not valid. Specifically, the Air Force’s Assistant Deputy Chief  of Staff, Installations and Logistics, informed the program office in a  September 2002 correspondence that it would not use the system, stating  that the Air Force would rely instead on its existing system capabilities.  Similarly, in March 2004, the Marine Corps’ Deputy Commandant for  Installations and Logistics informed the program office that the Corps  would not use the system because of concerns about whether the system  would, among other things, deliver expected capabilities. Program officials  told us that the latest economic analysis includes the costs and benefits for  all services because, despite communications to the contrary from the Air  Force and Marine Corps, the program office still considers TC-AIMS II to  be a multiservice program and has not yet been directed to assume and  plan otherwise.
In addition to the unreliable cost and benefit estimates, the latest analysis  does not show a benefit-to-cost ratio that is greater than 1. Specifically, the  analysis identified present value, total expected benefits of $928 million  versus present value, and total estimated costs of $929 million, for a  benefit-to-cost ratio of slightly less than 1. This means that for each dollar  spent on the system, slightly less than one dollar of benefit is returned. In  addition, the analysis cited various qualitative benefits (e.g., increasing  commander visibility of assets and enhancing worker productivity) that  would be derived from the system. However, the analysis did not provide  any underlying support or analysis for these qualitative benefits.
Program officials said that they recognize that the system’s expected return  on investment is marginal, but they nevertheless have been directed by the  milestone decision authority—the Assistant Secretary for Defense,  Networks and Information Integration—to deliver the system’s capabilities  as planned. According to the officials in the Assistant Secretary’s office, the  decision to continue investing in TC-AIMS II was based on quantitative and  qualitative benefits, which they stated together exceeded costs, thus  making the investment justified. However, these officials did not provide  supporting justification for their qualitative benefit claims. In addition,  their position does not recognize the intended change in the program’s  scope that is not reflected in the latest analysis. Thus, the Army lacks the  basis for knowing whether its planned investment in TC-AIMS II is justified.
section_title: Investments Have Not Been Incrementally Justified
paragraphs: DOD policy and guidance, as well as other related federal and best practice  guidance, state that large projects, such as TC-AIMS II, should be divided  into a series of smaller, incremental subprojects or releases so that  investment decisions can be made on each increment. By doing this, the  tremendous risk associated with investing large sums of money over many  years in anticipation of delivering system capabilities and associated  expected business value far into the future can be spread across project  increments that are smaller, of shorter duration, and capable of being more  reliably justified and more effectively managed against cost, benefit, and  risk expectations.
The Army has neither analyzed TC-AIMS II costs and benefits, nor made  associated investment decisions, on an incremental basis. Specifically,  while the program office divided the system’s acquisition into five smaller  increments (blocks), the economic analyses addressed life-cycle costs and  benefits for the entire system. The analyses did not identify separately the  costs and benefits for each of the five increments, and, thus, this  information was not considered as program officials made decisions about  TC-AIMS II at key milestones. Program officials stated that they have not  analyzed costs and benefits on an incremental basis because once the  program had been approved and initiated, they did not see the need  thereafter to justify each increment. By not determining the costs and  benefits of each of the system’s increments, the Army runs the risk of  discovering too late (i.e., after it has invested hundreds of millions of  dollars) that TC-AIMS II is not cost-beneficial.
section_title: TC-AIMS II Has Not Been Defined and Developed within the Context of a DOD Enterprise Architecture
paragraphs: DOD’s acquisition policies and guidance, as well as federal and best  practice guidance, recognize the importance of investing in IT business  systems within the context of an enterprise architecture. Our research and  experience in reviewing federal agencies show that not doing so often  results in systems that are duplicative, are not well integrated, are  unnecessarily costly to interface and maintain, and do not optimally  support mission outcomes. TC-AIMS II has not been defined and  developed in the context of a DOD enterprise architecture. Instead, the  Army has pursued the system on the basis of an Army logistics-focused  architecture. This means that TC-AIMS II as a DOD-wide program is based  on a service-specific architecture and not a DOD-wide architecture, thus  increasing the risk that TC-AIMS II, as defined, will not properly fit within  the context of future DOD enterprisewide business operations and IT  environments.
subsections: section_title: Well-defined DOD Enterprise Architecture to Guide and Constrain TC-AIMS II Has Not Existed
paragraphs: A well-defined enterprise architecture provides a clear and comprehensive  picture of an entity, whether it is an organization (e.g., a federal  department) or a functional or mission area that cuts across more than one  organization (e.g., personnel management). This picture consists of  snapshots of both the enterprise’s current or “As Is” environment and its  target or “To Be” environment, as well as a capital investment road map for  transitioning from the current to the target environment. These snapshots  consist of integrated “views,” which are one or more architecture products  that describe, for example, the enterprise’s business processes and rules;  information needs and flows among functions, supporting systems,  services, and applications; and data and technical standards and structures.
DOD has long operated without a well-defined enterprise architecture for  its business environment. In 2001, we first reported that DOD did not have  such an architecture, and we recommended that it develop one to guide  and constrain IT business systems, such as TC-AIMS II. Over the next 4  years, we have reported that DOD’s architecture development efforts were  not resulting in the kind of business enterprise architecture that could  effectively guide and constrain business system investments, largely  because the department did not have in place the architecture management  structures and processes described in federal guidance. In particular, we  most recently reported in July 2005 that despite spending about $318  million producing eight versions of its architecture, DOD’s latest version  still did not have, for example, a clearly defined purpose that could be  linked to the department’s goals and objectives, and a description of the  “As Is” environment and a transition plan. Further, we reported that the  description of the “To Be” environment was still missing important content  (depth and scope of operational and technical requirements) relative to, for  example, the actual systems to be developed or acquired to support future  business operations and the physical infrastructure (e.g., hardware and  software) that would be needed to support the business systems. Over the  last several years, we have also reported that DOD’s efforts for determining  whether ongoing investments were aligned to its evolving architecture  were not documented and independently verifiable. On September 28,  2005, DOD’s Under Secretary of Defense for Acquisition, Technology, and  Logistics and Under Secretary for Defense (Comptroller) issued the next  version of its business enterprise architecture, which we are required to  review, along with other things, such as the department’s efforts to review  certain investments’ alignment with the architecture, pursuant to the Fiscal  Year 2005 National Defense Authorization Act.
According to program officials, the system has not been assessed against  DOD’s business enterprise architecture because one has yet to be  completed. Instead, as described in more detail below, program officials  told us they have aligned TC-AIMS II with the Army’s enterprise  architecture for logistics. However, without a well-defined architecture that  sets the DOD-wide context within which a DOD-wide system is to fit, and a  firm understanding of the extent to which TC-AIMS II, as defined, fits  within that context, the program office risks acquiring a system that does  not meet DOD business needs and introduces redundancies and  incompatibilities that require costly and time-consuming rework to fix.
section_title: Army Reports That TC-AIMS II Is Aligned with an Army-Specific Architecture
paragraphs: In the absence of a DOD-wide architecture, program officials told us they  have developed the system in the context of the Army’s enterprise  architecture for logistics—the Single Army Logistical Enterprise, which is  managed by the Army’s Chief Information Officer. For example, in 2004,  program officials requested the Army’s Chief Information Officer to  determine whether TC-AIMS II was aligned with the logistic architecture.  The Office of the Chief Information Officer conducted an analysis during  the fall of 2004 that focused on, among other things, tracing TC-AIMS II  business processes and requirements to those specified in the logistics  architecture. The Chief Information Officer reported in March 2005 that  TC-AIMS II, as defined in Blocks 1 to 3, was aligned with the logistics  architecture. We did not review, and thus do not question, this analysis  because TC-AIMS II is intended to be a joint system, and thus aligning it  solely with the Army’s logistics architecture will not provide the program  with sufficient basis for managing the risk of not being able to adequately  ensure that a DOD-wide system aligns with a DOD-wide architecture.
section_title: Risk Management Has Not Been Effectively Implemented
paragraphs: Effective risk management is vital to the success of any system acquisition.  Accordingly, DOD acquisition management policies and guidance, as well  as other relevant best practice guidance, advocate proactively identifying  facts and circumstances that can increase the probability of an acquisition’s  failing to meet cost, schedule, and performance commitments and then  taking steps to reduce the probability of their occurrence and impact.  Effective risk management includes   developing written policies and procedures;   assigning roles and responsibilities for managing risk;   developing a risk management plan that provides for (1) identifying and  prioritizing risks, (2) developing and implementing the appropriate risk  mitigation strategies, and (3) tracking and reporting on progress in  implementing the strategies; and   executing the plan.
To its credit, the program office has satisfied the first three of these four  key practices. First, it has adopted and uses the risk policies and  procedures specified in DOD’s major IT systems acquisition guidance.  Second, the program office has assigned risk management roles and  responsibilities to the Government Risk Manager, who chairs the program’s  Risk Management Board. The board, which consists of key program  managers and stakeholder representatives, is responsible for ensuring  that the risk management program is implemented according to the risk  management plan. Among other things, this includes assigning individuals  who are to develop and implement risk mitigation plans for all newly  identified risks. Third, the program office has a risk management plan,  dated October 2004, and this plan provides for (1) identifying and  prioritizing risks, (2) developing and implementing the appropriate  mitigation strategies, and (3) tracking and reporting on progress in  implementing the strategies. For example, the plan defines procedures for  program staff and the board to follow in prioritizing risks, including  assessing its “risk exposure” on the basis of a probability of occurrence and  potential impact. On the basis of the assessment, “risk exposure” is  designated as either high, medium, or low. As of May 2005, the TC-AIMS II  program office had identified 22 active risks in its risk inventory with the  following exposures—1 high, 18 medium, and 3 low.
Fourth, since the program office has not fully implemented its risk  management plan, it has not satisfied the fourth risk management  practice—execution. For example, of the 22 active risks, 6 did not have  mitigation strategies. Of these 6, 1 is high exposure and 5 are medium  exposure risks. As another example, the program office’s inventory does  not include all key risks, such as the lack of reliable economic justification  and alignment to a DOD-wide architecture as previously described in this  report.
Program office officials stated that these risk management practices have  not been fully implemented because of other competing priorities. Further,  officials attributed the lack of risk mitigation strategies to staff turnover,  which resulted in relatively few staff receiving training on how to use the  automated tool employed by the office to document and track risks. In  particular, program officials stated that only about 20 percent of the  program staff (40 individuals) has attended the program’s training on how  to use the automated tool.
When we concluded our audit work, program officials acknowledged the  missing risk mitigation strategies and stated that they are in the process of  addressing them. Until the program fully implements effective risk  management practices, there is increased probability that program risks  will become actual problems leading to program cost, schedule, and  performance shortfalls.
section_title: System Requirements Are Being Effectively Managed
paragraphs: DOD policy and guidance, as well as other relevant best practices,  recognize the importance of effectively developing and managing system  requirements. According to the policy and guidance, well-defined  requirements are important because they establish agreement among the  various stakeholders on what the system is to do, how well it is to do it, and  how it is to interact with other systems. Having this agreement is key to  acquiring a system that meets end-user needs and performs as intended.
Effective requirements development and management involve, among  other things, (1) establishing a written policy on establishing and  conducting requirements development and management activities; (2)  eliciting desired and required system capabilities from users and  translating them into system requirements; (3) documenting the  requirements, including having users validate that they have been  accurately captured; (4) ensuring that requirements changes are controlled  as the system is developed and implemented; and (5) maintaining  bidirectional traceability, meaning that a given requirement can be traced  backward to its source and forward to both the component or increment  that will satisfy the requirement and the test that will verify that it is  satisfied.
The program office is managing TC-AIMS II requirements in accordance  with these five practices. First, the program office adopted DOD  Instruction 5000.2 and related guidance, as the program’s policy for  requirements development and management activities. The directive and  guidance are also incorporated in program office plans that detail how the  office develops and manages requirements during their life cycle—that is,  from when requirements are first documented to when system  responsibility is transferred to operations and maintenance support.  According to program officials responsible for managing requirements,  they use the guidance and plans to perform their work.
Second, program officials elicited required system capabilities from end  users through the program’s Joint Requirements Board, which consists of  key system stakeholders and representatives and is chaired by the Joint  Forces Command. The board is responsible for managing requirements  development activities. Among other things, the board’s charter states that  its duties include defining, reviewing, validating, prioritizing, and approving  the system’s requirements. Board meeting minutes show evidence of it  performing these duties, including, for example, reviewing the Block 3  baseline requirements document on September 8, 2004.
Third, program officials showed us how they document the requirements,  using an automated tool. Further, the Joint Requirements Board approved  the requirements baseline for each of the three blocks.
Fourth, to manage system requirement changes, the program office  established a change control structure (configuration control board) and  supporting processes, which are documented in its 2004 configuration  management plan. Consistent with the plan, the board has six voting  members and includes representatives from the four services. As shown in  board meeting minutes and supporting documentation, the board meets  monthly to consider and decide on change requests.
Fifth, the program maintains bidirectional traceability for the requirements  using an automated tool. Specifically, program officials demonstrated to  us how each requirement can be traced back and to its source (e.g.,  TC-AIMS II’s operational requirements document) and forward to the block  in which the requirement is to be satisfied.
section_title: Important Steps Have Been Taken to Effectively Leverage Commercial Components
paragraphs: The quality of the processes and practices followed in acquiring software- intensive systems greatly influences the quality of the systems produced.  Moreover, acquiring custom-developed system solutions is sufficiently  different from acquiring commercial component-based systems that  adherence to certain practices unique to the latter is key to their success.  As we have previously reported, DOD’s revised systems acquisition  policies and guidance require that commercial components be used to the  maximum extent that is feasible. Collectively, they also provide for the  following commercial component acquisition management best practices:  modifying components only after a thorough analysis of life-cycle costs   ensuring that integration contractors are explicitly evaluated on their  ability to implement commercial components, and  ensuring that project plans provide for the time and resources needed to  integrate commercial components with existing (legacy) systems.
The Army has largely complied with these practices for TC-AIMS II. First,  program officials stated that it is their policy to discourage component  modifications, which they have communicated in meetings and other  program management directives. In addition, the program has established a  joint configuration control board and supporting processes to limit and  control changes to the commercial products being used. As stated in the  program’s configuration management plan (dated June 30, 2005), the board,  which consists of key program managers and stakeholder representatives,  only approves proposed changes to those products if the change is justified  by a thorough analysis of costs, benefits, and risks. The board meets  monthly to discuss and decide upon such changes, and, according to  program officials, this approach has resulted in no major modifications to  date.
Second, the program office evaluation of competing contractors included  consideration of their ability to implement commercial components. For  example, the TC-AIMS II request for proposal specified that the  contractor’s ability to implement commercial products was to be a  significant factor in evaluating overall performance.
Third, the program’s work breakdown schedule (dated Nov. 9, 2004)  specifies the time and resources needed for integrating commercial  components with existing systems.
By following these best practices associated with acquiring commercial- based systems, the program is increasing the likelihood that TC-AIMS II  will be successfully implemented and effectively used.
section_title: Effective Contractor Management Practices Have Not Been Fully Employed
paragraphs: DOD policy and guidance and other relevant system acquisition  management best practices recognize the importance of effectively  managing contractor activities. To this end, policy and guidance call for,  among other things, performing contractor tracking and oversight activities  and using performance-based contracts to the maximum extent  practicable. To its credit, the Army has performed key activities related to  tracking and oversight. However, it has not fully implemented one key  aspect of performance-based contracting. By not doing so, the program  office has limited its ability to effectively influence contractor  performance.
subsections: section_title: Effective Contractor Tracking and Oversight Practices Have Been Followed
paragraphs: Department policy and guidance and related best practices guidance  identify effective contractor tracking and oversight as a key contract  management activity and define a number of associated practices to follow,  including  establishing a written policy on contract tracking and oversight;  designating responsibility for contract tracking and oversight activities  and having contracting specialists perform these activities; and  using approved contractor planning documents as part of periodic  reviews and interchanges with the contractor, during which actual cost  and schedule performance is compared with the contractor’s planned  budgets and schedules to identify variances and issues.
The program office’s tracking and oversight of the systems integration  contractor from Block 3 largely satisfies these three practices. First, the  program office has adopted the department’s acquisition management  polices and guidance as its written program-specific policy on contractor  tracking and oversight.
Second, the program office assigned responsibility for contract tracking  and oversight to its Budget Management Division, which has three staff  devoted to these duties. The devoted staff are contracting specialists on  detail from the General Services Administration’s Federal Systems  Integration and Management Center.
Third, the program office uses approved contractor planning documents as  the basis for overseeing the contractor. These planning documents are  specified in the contract and include a program management plan, a master  project schedule, and program status reports. Collectively, these  documents delineate when and how products and services are to be  delivered. According to program officials, these planning documents are  used during monthly status meetings with the contractor to track progress.  In particular, they are used to compare actual cost and schedule  performance to date with planned budgets and schedules to identify  potential variances.
Program officials attribute the state of their contract tracking and oversight  capability to two factors. First, the program has been in existence for about  10 years, thus allowing time for the office to develop disciplined processes  in this area. Second, in hiring the Federal Systems Integration and  Management Center contract specialists, the program has subject matter  experts who know the importance of, and who actively advocate,  rigorously following the processes. By employing key contract tracking and  oversight practices, the program office is increasing the likelihood that the  contractor will perform as expected.
section_title: Performance-Based Contracting Approach Is Not Fully Consistent with Policy and Best Practices
paragraphs: Department policy, federal acquisition regulations, and other relevant best  practices also advocate the use of performance-based contracting when  acquiring services. Effective performance-based contracting includes  defining clearly the work to be performed,  specifying performance standards (quality and timeliness) that are tied   establishing positive and negative incentives that are tied to the  contractor’s performance, and  having a quality assurance plan that describes how the contractor’s  performance in meeting requirements will be measured against  standards.
The program office’s approach to managing the contract is largely  consistent with these four practices. First, the contract for Block 3 (dated  Apr. 30, 2004) includes a statement of objectives that define the work to be  performed. For example, the contract identifies specific project tasks (e.g.,  establishing a Block 3 development help desk for system users) and  associated project management products (e.g., program management plan,  program schedule, and program status reports) that the contractor is to  perform and deliver, respectively, in accordance with the contract.
Second, the contract includes performance standards (measures) that are  contractually required. For example, it states that the help desk is to be  staffed 24 hours per day, 7 days a week, with help operators capable of  resolving 80 percent of user calls within 1 hour, while limiting caller on- hold time to 3 minutes or less.
Third, the contract provides positive incentives that are tied to contractor  performance. That is, in addition to the base contract (cost plus fixed fee)  amount, the contract includes an additional 8 percent fee that can be  awarded to the contractor for satisfying specified performance criteria.
Fourth, the latest quality assurance plan, dated August 2004, describes how  the contractor’s performance will be measured and what remedial steps are  to be taken in case the contractor does not meet contractual requirements.
However, the contract does not specify effective disincentives or penalties  for failing to meet the performance criteria. More precisely, the contract  does not provide for forfeiting all or part of the fee if the contractor fails to  perform as expected. Instead, program officials stated that their practice of  withholding the additional award fee when expectations are not met,  constitutes a disincentive. For example, they said that they withheld the  award fee for the contractor’s first performance period (which totaled  about $900,000) because the contractor failed to develop two deliverables  on time (the program management plan and the performance metrics).
We do not agree that this practice constitutes an effective negative  incentive because it does not penalize performance that fails to meet  expectations. Moreover, program officials told us that in the above  example, withholding the award fee was not a forfeiture, because it was  actually deferred to the second performance period when the contractor  had another opportunity to earn it. According to program officials, during  the second performance period, the contractor earned almost $780,000 of  the withheld fee from the first period by completing, among other things,  first period tasks.
In the absence of specific negative incentives that are tied to intended  performance, the program is at increased risk of contractor  nonperformance. This practice puts the program at risk of taking more  time and spending more money than necessary to acquire contract  deliverables and services.
section_title: Conclusions
paragraphs: It is unclear whether the Army’s planned investment in TC-AIMS II is  warranted. Of critical concern is the absence of reliable analysis showing  that further investment will produce future mission benefits commensurate  with estimated costs. This is due in large part to the evident change in the  scope of the program, which was initiated on the basis of the four military  services using TC-AIMS II but has not been revised to reflect that two  services have stated their intentions not to use the system. Compounding  this uncertainty is the inherent risk of defining and developing this  multiservice system outside the context of a well-defined DOD-wide  enterprise architecture. Without this information, the Army cannot  currently determine whether TC-AIMS II, as defined and as being  developed, is the right solution to meet its strategic business and  technological needs.
If these uncertainties are addressed and the Army demonstrates that  TC-AIMS II plans are the right course of action, the department largely has  the capabilities that are essential to successful system acquisition and  deployment, although there is room for strengthening its practices related  to risk management and performance-based contracting.
It is vital that Army and DOD authorities responsible and accountable for  ensuring prudent use of limited resources first reassess whether allowing  TC-AIMS II to continue as planned is warranted, and that the decision on  how to proceed be based on reliable data about program costs, benefits,  risk, and status.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the Secretary of the  Army to determine if continued investment in TC-AIMS II as planned  represents a prudent use of the department’s limited resources. To  accomplish this, the Secretary of the Army should take the following three  actions:  collaborate with the Office of the Assistant Secretary of Defense for  Networks and Information Integration/Chief Information Officer, the  Office of Program Analysis and Evaluation, and the Army Cost Analysis  Division to prepare a reliable economic analysis;  ensure that development of this economic analysis (1) complies with  cost-estimating best practices and relevant Office of Management and  Budget cost benefit guidance, (2) incorporates available data on  whether deployed TC-AIMS II capabilities are actually producing  benefits, and (3) addresses that the Air Force and Marines are not  planning to use the system; and  collaborate with the Undersecretary of Defense for Acquisition,  Technology, and Logistics and the Under Secretary of Defense  (Comptroller) to ensure that TC-AIMS II is adequately aligned with the  evolving DOD business enterprise architecture.
In addition, we recommend that the Secretary of Defense direct the  Secretary of the Army to present the results of these analyses to the Deputy  Secretary of Defense, or his designee, and seek a departmental decision on  how best to proceed with the program. Until this is done, we recommend  that the Secretary of Defense direct the Secretary of the Army to limit  future investment in already deployed applications to essential operation  and maintenance activities and only developmental activities deemed  essential to national security needs.
If—on the basis of reliable data—a decision is made to further develop  TC-AIMS II, we recommend that the Secretary of Defense direct the  Secretary of the Army to ensure that the program implements effective  program management activities related to risk management and  performance-based contracting.
section_title: Agency Comments and Our Evaluation
paragraphs: In its written comments on our draft report, signed by the Deputy to the  Assistant Secretary of Defense for Networks and Information Integration  (Command, Control, Communications, Intelligence, Surveillance, and  Reconnaissance and Information Technology Acquisition) and reprinted in  appendix III, DOD either agreed or partially agreed with our  recommendations, and it stated that some of our findings have merit and  that it has significant efforts ongoing to comply with applicable guidance.  For example, while DOD agreed that TC-AIMS II was originally defined and  implemented without a complete and formal enterprise architecture and  that the program had not identified the lack of a mature architecture as a  program risk, it stated that the program is taking steps to ensure alignment  to relevant enterprise architectures and is thereby mitigating this risk. In  addition, although the department agreed that the latest economic analysis  shows a negative return on investment, it said that other intangible benefits  not included in the analysis, such as improved planning and equipment  tracking, bolster the value of TC-AIMS II. Further, it agreed that the system  was originally envisioned as a single DOD-wide solution and that the Air  Force and Marine Corps are no longer intending to use the system, but  added that it was now evaluating whether these two services’ existing  (legacy) systems should be enabled to share data with TC-AIMS II, rather  than be replaced in their entirety by a joint system as was originally  envisioned. According to DOD, the results of this evaluation are to be  considered at the next milestone review, which is planned for early to mid- 2006.
These comments are largely consistent with our report. Specifically, the  report makes clear that DOD is managing TC-AIMS II in accordance with  some key practices and has efforts under way, such as those previously  cited, intended to bring the program into compliance with other key  practices. However, the report also points out noncompliance in areas  intended to provide sufficient information to determine how the TC-AIMS  II program should proceed. Accordingly, it contains recommendations to  ensure that the economic analysis is updated to (among other things)  reflect current operating assumptions, and that the program is adequately  aligned with the department’s evolving enterprise architectures. It is  precisely because the program does not have this information that, despite  having invested $751 million over 11 years, the department does not know  whether the program as defined is the right solution to meet DOD-wide  asset deployment needs.
DOD offered five detailed comments on our four recommendations. The  department’s comments and our responses are as follow: 1. DOD stated that the TC-AIMS II program has used best practices and  followed all economic analysis requirements specified in relevant DOD  and OMB guidance, and has successfully passed milestone reviews at  which it was required to show costs and benefits for both the entire  system and its increments. In addition, the department stated that even  though its economic analysis showed a slight negative return on  investment, the system’s intangible benefits bolster the program’s  overall value. According to DOD, the program identified these  intangible benefits to obtain department milestone approval, and this  documentation was provided to us on October 31, 2005. This  notwithstanding, the department stated the program will (1) conduct,  in coordination with the Office of the Secretary of Defense’s Program  Analysis and Evaluation, an analysis to determine the relationship of  TC-AIMS II and other programs, such as the Global Combat Support  System, that may provide similar functionality to ensure optimal return  on investment and (2) brief the Networks and Information Integration  Overarching Integrated Product Team in preparation for the next major  milestone decision review. It also stated that these results will be used  to justify further investment.
We do not agree that the program has used all best practices and  followed all requirements specified in relevant guidance. In particular,  the Army did not economically justify its TC-AIMS II investment on the  basis of reliable estimates of costs and benefits and did not invest in  TC-AIMS II within the context of a well-defined architecture, both of  which are recognized best practices as well as DOD and OMB  requirements. In addition, while it is appropriate to consider intangible  benefits in economically justifying an investment, these benefits are not  in the economic analysis, and the separate document provided on  October 31, 2005, did not contain sufficient rationale and justification  for claimed qualitative benefits. Rather, the information provided was a  restatement of the quantitative benefits in the economic analysis.  Further, by stating that it is to conduct an analysis of the relationships  of TC-AIMS II to other programs that provide similar functionality and  use the results of this analysis to justify further system development,  DOD is recognizing that it does not have the information it needs to  make informed investment decisions. If done thoroughly, the analysis,  which the department committed to conduct, should help to inform  future program decisions.  2. Consistent with our report, DOD stated that TC-AIMS II was  established as a joint military service program. However, in explaining  why the Air Force and Marine Corps are not intending to use TC-AIMS  II, it added that changes to operations throughout the world since then  have necessitated changes to the services’ deployment processes. In  light of this, the department stated it will evaluate other options for  data exchange among Air Force and Marine Corps existing asset  deployment systems and TC-AIMS II, instead of having all the services  use a single, joint system. Further, the department said it will have the  program, in coordination with Program Analysis and Evaluation,  develop an economic analysis for TC-AIMS II that is based on the  system being used solely by the Army and the Navy. DOD also  commented that our report assumed a positive cost-benefit impact  from including the Air Force and Marine Corps in the program.
We support the department’s decision to develop an economic analysis  that reflects this material change in the program’s scope, which is what  we recommended. However, we would note that DOD’s comments do  not explain how unspecified changes to operations throughout the  world caused unspecified changes to each of the services’ deployment  processes, which in turn required DOD to have to interface multiple  service-unique systems, rather than adopting a joint system. We would  expect such an explanation to be an integral part of updating the  economic analysis, as will the costs of pursuing service-unique courses  of action.
With regard to the comment that we assumed the inclusion of the Air  Force and Marine Corps would have a positive cost-benefit impact, we  do not agree. We made no assumption about the impact of the  program’s scope on the cost-to-benefit or return on investment ratios.
Rather, our point is that DOD policy and related guidance state that  programs should be economically justified by, among other things, cost  and benefit estimates that are based on realistic plans for the program’s  scope. Such realistic plans did not exist because DOD’s justification  was based on the four services using the system, when in reality the Air  Force and Marine Corps do not intend to do so. Relevant guidance calls  for updating economic analyses when significant program changes  such as these occur. Accordingly, we recommended that the  department follow this guidance to ensure it had adequate information  for informed decision making. 3. DOD commented that it has worked over the past several years to  develop an enterprise architecture and is on a path to fully defining and  effectively managing its enterprise architecture. It also said that it is  working to align TC-AIMS II with relevant evolving architectures. In  particular, DOD stated that Blocks 1 through 4 are aligned with the  Joint Deployment Operational Architecture and the program office  continues to work to align the program with the Joint Deployment and  Distribution Architecture, which it said is currently aligning to the  evolving Business Enterprise Architecture.
We do not take issue with the comments that work has been and is  under way in this area. Our point is that the Business Enterprise  Architecture is still a work in progress and what has been available to  guide and constrain business system investments, such as TC-AIMS II,  has not been sufficient. Moreover, program officials told us that  TC-AIMS II has not been assessed against the Business Enterprise  Architecture because this architecture has not yet been completed.  Without a well-defined DOD-wide architecture that sets the context  within which such a joint military service system is to fit, and a firm  understanding of the extent to which TC-AIMS II, as defined, fits within  that context, the program office risks, among other things, acquiring a  system that does not meet DOD’s strategic business needs.  4. DOD stated that while comprehensive mitigation and contingency  strategies were not developed for all of the 22 risks cited in our report,  the program office has subsequently developed strategies for them.  DOD also commented that subsequent risk management data, along  with evidence that risks are continuously reviewed and updated, was  provided to us on October 28 and November 3, 2005.
We acknowledge that program officials orally stated that this had  occurred. However, we have yet to receive any documentation to  substantiate these statements.  5. DOD stated that while the current TC-AIMS II contract does not include  direct disincentives for contractor failure to meet performance criteria,  its semiannual award fee mechanism serves a similar purpose.  Nonetheless, DOD commented that it will address its award fee  implementation on future TC-AIMS II contract reviews.
We do not agree that this semiannual award fee approach is an effective  disincentive because award fees are only being withheld when  contractor performance does not exceed certain targets; conversely,  direct disincentives penalize poor performance that falls short of  meeting contractually required levels of performance. Without direct  disincentives, the program is at increased risk of contractor  nonperformance, and thus taking more time and spending more money  than necessary to acquire contract deliverables and services. We  support the department’s stated commitment to address this issue.
We are sending copies of this report to interested congressional  committees. We are also sending copies to the Director, Office of  Management and Budget; the Secretary of Defense; the Deputy Secretary of  Defense; the Undersecretary of Defense for Acquisition, Technology, and  Logistics; the Assistant Secretary of Defense (Networks and Information  Integration)/Chief Information Officer; the Secretaries of the Air Force,  Army, and Navy; the Commandant of the Marine Corps; the Department of  the Army’s Chief Information Officer; and the Program Executive Officer,  Enterprise Information Systems, Transportation Information Systems,  Department of the Army. We are also sending copies to other interested  parties. This report will also be available at no charge on our Web site at  http://www.gao.gov.
Should you have any questions about matters discussed in this report,  please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. A GAO contact and staff who made major  contributions to this report are listed in appendix IV.
section_title: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the Transportation Coordinators’  Automated Information for Movements System II (TC-AIMS II) was being  managed according to important aspects of the Department of Defense’s  (DOD) acquisition policies and guidance, as well as other relevant  acquisition management best practices. To accomplish this, we focused on  the program’s (1) economic justification; (2) architectural alignment;  (3) risk management; (4) requirements development and management;  (5) commercial components; and (6) contract management, including  contractor oversight and performance-based contracting.
To determine whether the Department of the Army had economically  justified its investment in TC-AIMS II, we reviewed the latest economic  analysis to determine the basis for the cost and benefit estimates and net  present value calculations. This included evaluating the analysis against  DOD policies and guidance as well as other relevant best practices  guidance. It also included interviewing responsible program officials,  including the program and deputy program managers, regarding their  respective roles, responsibilities, and actual efforts in developing and/or  reviewing the economic analysis. In addition, we also interviewed the  program and deputy program manager and the Office of the Assistant  Secretary of Defense for Network Information and Integration about the  purpose and use of the analysis for managing the Army’s investment in the  TC-AIMS II program, including the extent to which measures and metrics  showed that benefits projected in the economic analysis were actually  being realized.
To determine whether the Army had aligned TC-AIMS II to the DOD  business enterprise architecture, we relied on our prior reports addressing  DOD and Army architecture development and implementation efforts and  documents on the Army’s efforts to align TC-AIMS II with its logistics  architecture (the Single Army Logistics Enterprise), as well as documents  on the Army’s efforts to align the logistics architecture with DOD’s business  enterprise architecture. We also interviewed officials from the TC-AIMS II  joint program management office, the Office of the Assistant Secretary of  Defense for Networks and Information Integration/Chief Information  Officer, and the Office of the Army’s Chief Information Officer about DOD  and Army architecture efforts and TC-AIMS II’s alignment to them.
To determine whether the Army was effectively managing key system  acquisition activities, namely risk management, requirements development  and management, commercial components, and contract management, we  did the following:  To assess risk management, we reviewed the program’s risk policies,  processes, management plan, and related documentation. We also  observed how the program used an automated tool (database) to  document and track risks, including developing risk mitigation  strategies. In addition, we analyzed a version of the database to  determine, among other things, the state of risk mitigation strategies. We  then compared these activities with DOD policy, guidance, and related  best practices to determine whether variances existed, and if so, why.  Moreover, we interviewed Army officials involved with risk  management, including the program manager and risk manager, to  discuss their roles and responsibilities for managing risk associated  with acquiring and implementing TC-AIMS II.
To assess requirements development and management capabilities, we  reviewed program documentation, such as the official list of  requirements, and system specifications and evaluated them against  these relevant best practices for several characteristics, including  traceability and prioritization. We observed program office staff trace  requirements to both higher level documents and lower level  specifications using an automated tool. We interviewed Army officials  involved in the requirements development process, including the  program and deputy program managers, to discuss their roles and  responsibilities for developing and managing requirements.
To evaluate the management of commercial components, we reviewed a  variety of documents, such as program and contractor studies of  commercial components’ products, as well as the program office’s  project plans, and contract task order documentation. Additionally, we  conducted interviews with the program officials, including the program  and deputy program managers, responsible for selecting the commercial  components for planned TC-AIMS II blocks, to determine whether the  commercial component products had been acquired in accordance with  DOD policy, guidance, and related best practices.
To assess contract management, we focused on the program office’s  contract tracking and oversight activities as well as its use of  performance-based contracting. For tracking and oversight, we  reviewed program management plans, contractor monthly status  reports, and related documentation of the program office’s policies,  processes, and activities for overseeing its system integration contractor  for Block 3. We then compared these with DOD policies, guidance, and  related best practices. We also interviewed program offices responsible  for managing these activities. For performance-based contracting, we  reviewed the Block 3 contract (issued in April 2004) and related  documentation, such as contract modifications, quality assurance plan,  and reports on determining contractor award fee, and then compared  them with department policy and guidance, federal acquisition  regulations, and related best practices. We also interviewed program  management officials, including the contract specialist from the General  Service Administration’s Federal Systems Integration and Management  Center, who are responsible for performing these activities.
We did not independently validate information on the program’s cost and  budget as well as the number of problem reports.
We conducted our work at DOD headquarters in the Washington, D.C.,  metropolitan area and at the Army’s Joint Program Management Office in  Northern Virginia. We performed our work from October 2004 through  October 2005, in accordance with generally accepted government auditing  standards.
section_title: Assessment of Problem Reports
paragraphs: One indicator of system quality is defect density, which can be measured in  a number of ways, including the trend in the number of system defects  being reported and resolved. The program office prepares problem reports  to document and prioritize system defects, and it established a database to  manage and track these reports. Further, the program office assigns  priority (i.e., criticality) levels to problem reports, with category 1 being the  most critical, and category 5 being the least critical. The program office’s  definition of each category is as follows: Category 1: The system or function does not work and the problem must  be addressed immediately, before any further work can be completed.
Category 2: The system or function may work, but the problem is severe  and must be addressed before the end of the current block.
Category 3: Essentially, this is a category 2 problem with a known solution  that would provide a temporary fix.
Category 4: A routine problem that does not pose a significant threat and  thus no immediate action is required.
Category 5: A minor problem that does not pose a significant threat to  system functionality.
As depicted in figure 2, the data show that the number of open (yet to be  resolved) problem reports over the first two development blocks has  decreased. In particular, the total number of open problem reports has  decreased by 321 (or 75 percent), from 426 in fiscal year 1999 to 105 at the  end of fiscal year 2004. During the same period, categories 1 and 2 problem  reports decreased by 216 (or about 100 percent), from 217 to 1. This  downward trend reversed for the less critical defects in fiscal year 2005.  Specifically, as of September 30, 2005, there were 534 total defects, 25 of  which were categories 1, 2, and 3. Program officials attributed this reversal  of categories 4 and 5 defects to current Block 3 testing activities and stated  that they anticipated the increase and are actively managing it. In addition,  they stated that while overall defects have recently increased, the number  of critical defects has increased only slightly, and program staff are  working to resolve these increases, with the goal of having permanent  solutions implemented before Block 3 is deployed, which is scheduled to  begin in October 2006.
section_title: Comments from the Department of Defense
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Gary Mountjoy, Assistant Director;  Justin Booth; Hal Brumm Jr.; Joanne Fiorino; Anh Le; Kush K. Malhotra;  Teresa M. Neven; Dr. Rona Stillman; Amos Tevelow; and Bill Wadsworth  made key contributions to this report.