DOD Business Systems Modernization: Recent Slowdown in Institutionalizing Key Management Controls Needs to Be Addressed
Published: 2009-05-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 1995, GAO has designated the Department of Defense's (DOD) business systems modernization program as high risk, and it continues to do so today. To assist in addressing DOD's business system modernization challenges, the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 (the Act) contains provisions that require the department to take certain actions and to annually report to its congressional committees on these actions. The Act also directs GAO to review each annual report. In response, GAO performed its fifth annual review of DOD's actions to comply with key aspects in the Act and related federal guidance. To do so, GAO reviewed, for example, the latest version of DOD's business enterprise architecture (BEA) and transition plan, investment management policies and procedures, and information in the department's business system data repositories.
section_title: What GAO Found
paragraphs: The pace of DOD's progress in defining and implementing key institutional modernization management controls has slowed compared with progress made in each of the last 4 years, leaving much still to be accomplished to fully implement the Act's requirements and related guidance. In particular, the corporate BEA continues to evolve and address previously identified missing elements, inconsistencies, and usability issues, but gaps still remain. For example, while the BEA now identifies information assurance laws, regulations, and policies, it still does not include business rules for all business processes. Further, little progress has been made in the last year in extending (i.e., federating) the BEA to the entire family of business mission area architectures, including using an independent verification and validation agent to assess the components' subsidiary architectures and federation efforts. The updated enterprise transition plan continues to identify systems and initiatives, but important elements are still missing, as are individual component plans. For example, while the plan provides a range of information, such as budgets and performance measures, for key enterprisewide and component-specific investments, it is missing information on identified investments. The fiscal year 2009 budget submission included some, but omitted other, key information about business system investments, in part because of the lack of a reliable comprehensive inventory of all defense business systems. Investment approval and accountability structures have been established for DOD and the Air Force, and related policies and procedures that are consistent with relevant guidance have been partially defined. However, these structures and processes are still lacking for the Navy. Business system investments costing over $1 million continue to be certified and approved, but these decisions are not always based on complete information. For example, key Navy investments have not fully demonstrated compliance with the department's BEA, and their economic justifications were not based on reliable estimates of cost and benefits. In addition, the information in DOD's authoritative repository of system investments that is used to make these decisions is not always accurate. Department officials attributed this slowdown in large part to pending decisions surrounding the roles, responsibilities, authorities, and relationships among key senior leadership positions, such as DOD's Deputy Chief Management Officer and the military departments' Chief Management Officers. Until DOD fully implements these long-standing institutional modernization management controls provided for under the Act, addressed in GAO recommendations, and otherwise embodied in relevant guidance, its business systems modernization will likely remain a high-risk program. As a result, it is important that the department act quickly to resolve pending decisions about key positions.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated  DOD’s business systems modernization program as high risk, and we  continue to designate it as such today. As our research on public and  private sector organizations shows, two essential ingredients to a  successful systems modernization program are having a well-defined  enterprise architecture and an effective institutional approach to  managing information technolo gy (IT) investments.
Accordingly, we made recommendations to the Secretary of Defense in  May 2001 that included the means for effectively developing an enterprise  architecture and establishing a corporate, architecture-centric approach to  investment control and decision making. Between 2001 and 2005, we  reported that the department’s business systems modernization program  continued to lack both of these, concluding in 2005 that hundreds of  millions of dollars had been spent on a business enterprise architecture  (BEA) and investment management structures that had limited value.  Accordingly, we made more explicit architecture and investment  management-related recommendations.
To further assist DOD in addressing these modernization management  challenges, Congress included provisions in the Ronald W. Reagan  National Defense Authorization Act for Fiscal Year 2005 (the Act) that  were consistent with our recommendations. More specifically, the Act  requires the department to, among other things, (1) develop a BEA,   (2) develop a transition plan to implement the architecture, (3) identify  systems information in its annual budget submission, (4) establish a  system investment approval and accountability structure, (5) establish an  investment review process, and (6) certify and approve any system  modernizations costing in excess of $1 million. The Act further requires  that the Secretary of Defense submit an annual report to congressional  defense committees on DOD’s compliance with certain requirements of  the Act not later than March 15 of each year from 2005 through 2009.  Additionally, the Act directs us to submit to these congressional  committees—within 60 days of DOD’s report submission—an assessment  of DOD’s actions to comply with these requirements.
As agreed with your offices, the objective of our review was to assess the  actions taken by DOD to comply with requirements of section 2222 of Title  10, U.S. Code. To accomplish this, we used our prior annual report under  the Act as a baseline, analyzing whether the department had taken actions  to comply with those requirements, related guidance, and our prior  recommendations that we previously identified as not yet addressed. In  doing this, we also relied on the results of relevant reports that we have  issued since our prior annual report. We also reviewed the department’s  report to Congress, which was submitted on March 18, 2009, and evaluated  the information used to satisfy the budget submission and investment  review, certification, and approval aspects of the Act.
We conducted this performance audit at DOD offices in Arlington,  Virginia, from January to May 2009, in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.  Details on our objective, scope, and methodology are contained in  appendix I.
subsections: section_title: Background
paragraphs: DOD is a massive and complex organization and is entrusted with more  taxpayer dollars than any other federal department or agency. To  illustrate, Congress provided DOD with about $512 billion in  appropriations for fiscal year 2009. Additionally, Congress has provided  about $808 billion in supplemental emergency funding for operations in  support of the Global War on Terrorism since 2001. Moreover, the recent  American Recovery and Reinvestment Act of 2009 contains nearly $12.6  billion in appropriations for DOD for military construction, environmental  restoration, and other purposes. Organizationally, the department includes  the Office of the Secretary of Defense, the Joint Chiefs of Staff, the  military departments, numerous defense agencies and field activities, and  various unified combatant commands that are responsible for either  specific geographic regions or specific functions. (See fig. 1 for a  simplified depiction of DOD’s organizational structure.)
In support of its military operations, the department performs an  assortment of interrelated and interdependent business functions,  including logistics management, procurement, health care management,  and financial management. As we have previously reported, the DOD  systems environment that supports these business functions is overly  complex and error prone, and is characterized by (1) little standardization  across the department, (2) multiple systems performing the same tasks,  (3) the same data stored in multiple systems, and (4) the need for data to  be entered manually into multiple systems. Moreover, the department  recently reported that this systems environment is composed of  approximately 2,480 separate business systems. For fiscal year 2009, DOD  requested about $15.3 billion in funds to operate, maintain, and modernize  these business systems and associated IT infrastructure.
As we have previously reported, the department’s nonintegrated and  duplicative systems impair its ability to combat fraud, waste, and abuse.  In fact, DOD currently bears responsibility, in whole or in part, for 15 of  our 30 high-risk areas. Eight of these areas are specific to the  department, while it shares responsibility for 7 other governmentwide  high-risk areas. Collectively, these high-risk areas relate to DOD’s major  business operations that are inextricably linked to the department’s ability  to perform its overall mission, directly affect the readiness and capabilities  of U.S. military forces, and can affect the success of a mission. DOD’s  business systems modernization is one of the high-risk areas, and it is an  essential enabler to addressing many of the department’s other high-risk  areas. For example, modernized business systems are integral to the  department’s efforts to address its financial, supply chain, and information  security management high-risk areas.
subsections: section_title: Enterprise Architecture and IT Investment Management Controls Are Critical to Achieving Successful Systems Modernization
paragraphs: Effective use of an enterprise architecture—a modernization blueprint—is  a hallmark of successful public and private organizations. Since the early  1990s, we have promoted the use of architectures to guide and constrain  systems modernization, recognizing them as a crucial means to meeting  this challenging goal: optimally defined operational and technological  environments. Congress, the Office of Management and Budget (OMB),  and the federal Chief Information Officers (CIO) Council have also  recognized the importance of an architecture-centric approach to  modernization. The Clinger-Cohen Act of 1996 mandates that an agency’s  CIO develop, maintain, and facilitate the implementation of an information  technology architecture. Further, the E-Government Act of 2002 requires  OMB to oversee the development of enterprise architectures within and  across agencies. In addition, we, OMB, and the CIO Council have issued  guidance that emphasizes the need for system investments to be  consistent with these architectures. For example, in April 2003, we  issued a framework that emphasizes the importance of having an  enterprise architecture as a critical frame of reference for organizations  when they are making IT investment decisions. Also, in December 2008,  OMB issued guidance that addresses system investment compliance w ith  agency architectures.
A corporate approach to IT investment management is another important  characteristic of successful public and private organizations. Recognizing  this, Congress enacted the Clinger-Cohen Act of 1996, which requires  OMB to establish processes to analyze, track, and evaluate the risks and  results of major capital investments in IT systems made by executive  agencies. In response to the Clinger-Cohen Act and other statutes, OMB  has developed policy and issued guidance for planning, budgeting,  acquisition, and management of federal capital assets. We have also  issued guidance in this area that defines institutional structures (such as  investment boards), processes for developing information on investments  (such as cost/benefit), and practices to inform management decisions  (such as whether a given investment is aligned with an enterprise  architecture).
section_title: Enterprise Architecture: A Brief Description
paragraphs: An enterprise architecture provides a clear and comprehensive picture of  an entity, whether it is an organization (e.g., a federal department) or a  functional or mission area that cuts across more than one organization  (e.g., financial management). An architecture describes the enterprise in  logical terms (such as interrelated business processes and business rules,  information needs and flows, and work locations and users) as well as in  technical terms (such as hardware, software, data, communications,  security attributes, and performance standards). It provides these  perspectives both for the enterprise’s current, or “as is,” environment, and  for its target, or “to be,” environment, and it provides a transition plan for  moving from the “as is” to the “to be” environment. This transition plan  provides a temporal road map for moving between the two environments  and incorporates such considerations as technology opportunities,  marketplace trends, fiscal and budgetary constraints, institutional system  development and acquisition capabilities, legacy and new system  dependencies and life expectancies, and the projected value of competing  investments.
The suite of products produced for a given entity’s enterprise architecture,  including its structure and content, is largely governed by the framework  used to develop the architecture. Since the 1980s, various architecture  frameworks have been developed, such as John A. Zachman’s “A  Framework for Information Systems Architecture” and the DOD  Architecture Framework.
The importance of developing, implementing, and maintaining an  enterprise architecture is a basic tenet of both organizational  transformation and systems modernization. Managed properly, an  enterprise architecture can clarify and help optimize the  interdependencies and relationships among an organization’s business  operations and the underlying IT infrastructure and applications that  support these operations. Moreover, when an enterprise architecture is  employed in concert with other important management controls, such as  portfolio-based capital planning and investment control practices,  architectures can greatly increase the chances that an organization’s  operational and IT environments will be configured to optimize mission  performance. The alternative, as our work has shown, is the perpetuation  of the kinds of operational environments that burden many agencies  today, where a lack of integration among business operations and the IT  resources supporting them leads to systems that are duplicative, poorly  integrated, and unnecessarily costly to maintain and interface. Our  framework provides federal agencies with a common benchmarking tool  for planning and measuring their efforts to improve enterprise architecture  management.
One approach to structuring an enterprise architecture is referred to as a  federated enterprise architecture. Such a structure treats the architecture  as a family of coherent but distinct member architectures that conform to  an overarching architectural view and rule set. This approach recognizes  that each member of the federation has unique goals and needs as well as  common roles and responsibilities with the levels above and below it.  Under a federated approach, member architectures are substantially  autonomous, although they also inherit certain rules, policies, procedures,  and services from higher-level architectures. As such, a federated  architecture gives autonomy to an organization’s components while  ensuring enterprisewide linkages and alignment where appropriate. Where  commonality among components exists, there are also opportunities for  identifying and leveraging shared services.
A service-oriented architecture is an approach for sharing business  capabilities across the enterprise by designing functions and applications  as discrete, reusable, and business-oriented services. As such, service  orientation permits sharing capabilities that may be under the control of  different component organizations. As we have previously reported, such  capabilities or services need to be, among other things, (1) self-contained,  meaning that they do not depend on any other functions or applications to  execute a discrete unit of work; (2) published and exposed as self- describing business capabilities that can be accessed and used; and   (3) subscribed to via well-defined and standardized interfaces. A service- oriented architecture approach is thus intended not only to reduce  redundancy and increase integration, but also to provide the kind of  flexibility needed to support a quicker response to changing and evolving  business requirements and emerging conditions.
section_title: IT Investment Management: A Brief Description
paragraphs: IT investment management is a process for linking IT investment decisions  to an organization’s strategic objectives and business plans that focuses on  selecting, controlling, and evaluating investments in a manner that  minimizes risks while maximizing the return of investment.
During the selection phase, the organization (1) identifies and analyzes  each project’s risks and returns before committing significant funds to any  project and (2) selects those IT projects that will best support its mission  needs.
During the control phase, the organization ensures that as projects  develop and investment expenditures continue, they continue to meet  mission needs at the expected levels of cost and risk. If the project is not  meeting expectations, or if problems arise, steps are quickly taken to  address the deficiencies.
During the evaluation phase, actual versus expected results are compared  once a project has been fully implemented. This is done to (1) assess the  project’s impact on mission performance, (2) identify any changes or  modifications to the project that may be needed, and (3) revise the  investment management process based on lessons learned.
Consistent with this guidance, our IT Investment Management (ITIM)  framework consists of five progressive stages of maturity for any given  agency relative to selecting, controlling, and evaluating its investment  management capabilities. (See fig. 2 for the five ITIM stages of maturity.)  The overriding purpose of the framework is to encourage investment  selection and control and to evaluate processes that promote business  value and mission performance, reduce risk, and increase accountability  and transparency. We have used the framework in several of our  evaluations, and a number of agencies have adopted it.
In our ITIM framework, with the exception of the first stage, each maturity  stage is composed of “critical processes” that must be implemented and  institutionalized in order for the organization to achieve that stage. Each  ITIM critical process consists of “key practices”—to include organizational  structures, policies, and procedures—that must be executed to implement  the critical process. Our research shows that agency efforts to improve  investment management capabilities should focus on implementing all  lower-stage practices before addressing higher-stage practices.
Stage 2 critical processes lay the foundation by establishing successful,  predictable, and repeatable investment control processes at the project  level. Stage 3 is where the agency moves from project-centric processes to  portfolio-based processes and evaluates potential investments according  to how well they support the agency’s missions, strategies, and goals.  Organizations implementing these Stage 2 and 3 practices have in place  selection, control, and evaluation processes that are consistent with the  Clinger-Cohen Act. Stages 4 and 5 require the use of evaluation  techniques to continuously improve both investment processes and  portfolios in order to better achieve strategic outcomes.
section_title: DOD’s Institutional Approach to Business Systems Modernization
paragraphs: The National Defense Authorization Act (NDAA) for Fiscal Year 2008  designated the Deputy Secretary of Defense as the Chief Management  Officer (CMO) for DOD and created a Deputy CMO position. The CMO’s  responsibilities include developing and maintaining a departmentwide  strategic plan for business reform and establishing performance goals and  measures for improving and evaluating overall economy, efficiency, and  effectiveness and monitoring and measuring the progress of the  department. The Deputy CMO’s responsibilities include recommending to  the CMO methodologies and measurement criteria to better synchronize,  integrate, and coordinate the business operations to ensure alignment in  support of the warfighting mission. The Business Transformation Agency  (BTA) supports the Deputy CMO in leading and coordinating business  transformation efforts across the department.
The CMO and Deputy CMO are to interact with several entities to provide  executive leadership for the direction, oversight, and execution of DOD’s  business transformation efforts, which include business systems  modernization. These entities include the Defense Business Systems  Management Committee (DBSMC), which serves as the highest-ranking  investment review and decision-making body for business systems  modernization activities and is chaired by the Deputy Secretary of  Defense; the Principal Staff Assistants, who serve as the certification  authorities for business system modernizations in their respective core  business missions; the Investment Review Boards (IRB), which are  chaired by the certifying authorities and form the review and decision- making bodies for business system investments in their respective areas of  responsibility; and the BTA, which is responsible for supporting the IRBs,  and for leading and coordinating business transformation efforts across  the department. Table 1 lists these entities and provides greater detail on  their roles, responsibilities, and composition.
section_title: Tiered Accountability
paragraphs: In 2005, DOD reported that it had adopted a “tiered accountability”  approach to business systems modernization. Under this approach,  responsibility and accountability for business architectures and systems  investment management are assigned to different levels in the  organization. For example, the BTA is responsible for developing the  corporate BEA (i.e., the thin layer of DOD-wide policies, capabilities,  standards, and rules) and the associated enterprise transition plan (ETP).  The components are responsible for defining a component-level  architecture and transition plans associated with their own tiers of  responsibility and for doing so in a manner that is aligned with (i.e., does  not violate) the corporate BEA. Similarly, program managers are  responsible for developing program-level architectures and plans and  ensuring alignment with the architectures and transition plans above  them. This concept is to allow for autonomy while also ensuring linkages  and alignment from the program level through the component level to the  corporate level. Table 2 describes the four investment tiers and identifies  the associated reviewing and approving entities.
Consistent with the tiered accountability approach, the NDAA for Fiscal  Year 2008 required the Secretaries of the military departments to designate  the department Under Secretaries as CMOs with primary responsibility for  business operations. Moreover, the Duncan Hunter NDAA for Fiscal Year  2009 requires the military departments to establish business  transformation offices to assist their CMOs.
section_title: Summary of Fiscal Year 2005 NDAA Requirements
paragraphs: Congress included six provisions in the fiscal year 2005 NDAA that are  aimed at ensuring DOD’s development of a well-defined BEA and  associated ETP, as well as the establishment and implementation of  effective investment management structures and processes. The  requirements are as follows:  1.  Develop a BEA that includes an information infrastructure that, at a  minimum, would enable DOD to    comply with all federal accounting, financial management, and  routinely produce timely, accurate, and reliable financial information  for management purposes; integrate budget, accounting, and program information and systems;  and   provide for the systematic measurement of performance, including the  ability to produce timely, relevant, and reliable cost information.
In addition, the BEA must  include policies, procedures, data standards, and system interface  requirements that are to be applied uniformly throughout the  department and   be consistent with OMB policies and procedures.  2.  Develop an ETP for implementing the architecture that includes   an acquisition strategy for new systems needed to complete the    a list and schedule of legacy business systems to be terminated;    a list and strategy of modifications to legacy business systems; and  time-phased milestones, performance metrics, and a statement of  financial and nonfinancial resource needs. 3.  Identify each business system proposed for funding in DOD’s fiscal  year budget submissions and include    a description of the certification made on each business system  proposed for funding in that budget; funds, identified by appropriations, for current services and for  business systems modernization; and the designated approval authority for each business system. 4.  Delegate the responsibility for business systems to designated  approval authorities within the Office of the Secretary of Defense. 5.  Require each approval authority to establish investment review  structures and processes, including a hierarchy of IRBs—each with  appropriate representation from across the department. The review  process must include   a review and approval of each business system by an IRB before funds    at least an annual review of every business system investment;  the use of threshold criteria to ensure an appropriate level of review  and accountability; the use of procedures for making architecture compliance  certifications; the use of procedures consistent with DOD guidance; and  the incorporation of common decision criteria.  6.  Effective October 1, 2005, DOD may not obligate appropriated funds  for a defense business system modernization with a total cost of more  than $1 million unless the approval authority certifies that the business  system modernization    complies with the BEA; or  is necessary to achieve a critical national security capability or address  a critical requirement in an area such as safety or security, or is  necessary to prevent a significant adverse effect on an essential project  in consideration of alternative solutions; and the certification is approved by the DBSMC.
The fiscal year 2005 NDAA also requires that the Secretary of Defense  submit to the congressional defense committees a report on the  department’s compliance with the above provisions.
section_title: Summary of Recent GAO Reviews of DOD’s Business Systems Modernization and Business Transformation Efforts
paragraphs: Since 2005, we have reported that DOD has each year taken increasing  steps to comply with the requirements of the fiscal year 2005 NDAA and to  satisfy relevant systems modernization management guidance. Moreover,  we concluded that DOD had made important progress each year relative to  architecture development, transition plan development, budgetary  disclosure, and investment review; however, aspects of these requirements  and relevant guidance had yet to be fully satisfied. We also reported that  DOD had fully satisfied the requirement concerning designated approval  authorities and continued to certify and approve modernizations costing  more than $1 million. However, each report also concluded that much  remained to be accomplished relative to the Act’s requirements and  relevant guidance, as these examples illustrate:    The BEA lacked important content, such as business rules for, and  information flows among, certain business activities, and it had yet to be  extended (i.e., federated) throughout the DOD component organizations.
The ETP did not include investments for all components and did not  reflect key factors associated with properly sequencing planned  investments, such as dependencies among investments and the capability  to execute the plan.
DOD and the military departments had yet to fully establish key  investment review structures and define related policies and procedures  for effectively performing both project-level and portfolio-based  investment management.
Accordingly, we either provided new or reiterated existing  recommendations to address each of these areas. DOD largely agreed with  our recommendations. In August 2008, we also reported on issues with the  process used to certify investments as compliant with DOD’s BEA.  Specifically, we reported that key DOD business systems modernization  programs did not adequately demonstrate compliance with the  department’s federated BEA, even though each program had largely  followed DOD’s existing compliance guidance, used its compliance  assessment tool, and was certified and approved as being compliant by  department investment oversight and decision-making entities. In addition,  we reported that even though the department’s investment oversight and  decision-making authorities had certified and approved these business  system programs as compliant with the BEA, these certification and  approval entities did not validate each program’s compliance assessment  and assertions. Accordingly, we made recommendations to address each  of those shortcomings, which DOD agreed with.
With respect to departmentwide business transformation, we recently  reported that implementation of DOD’s overall management framework  for business transformation is not yet complete because key aspects had  not been defined. For example, we reported that the authority, roles, and  relationships for some positions and entities had not been clearly defined,  including a clearly defined decision-making authority for the Deputy CMO,  a clearly defined relationship between DOD’s Deputy CMO and the CMOs  of the military departments, and clearly defined unique and shared  responsibilities of various governance entities, such as the Deputy’s  Advisory Working Group and the DBSMC. We concluded that the current  administration needed to move quickly to nominate and fill key leadership  positions, including the Deputy Secretary of Defense (now statutorily  designated as the CMO), the Deputy CMO, the Under Secretaries of  Defense, and the military department CMOs. We also concluded that, in  light of the transition, it will be important for senior leaders in the current  administration to further define and clarify the roles, responsibilities, and  relationships among the various positions and governance entities within  DOD’s management framework for business transformation in order to  sustain and further DOD’s progress.
In addition, we reported that DOD’s first strategic management plan,  issued in 2008, lacked key information and elements necessary for  assisting in successfully achieving business management transformation.
For example, it did not identify any strategic goals, objectives, and  performance measures, and while it stated a purpose, the plan did not  provide detailed information about business operations. Without strategic  goals and objectives, we concluded that the strategic management plan  could not be linked to other existing plans and tools for individual  business areas, such as the ETP.
section_title: DOD Continues to Take Steps to Strengthen Management of Its Business Systems Modernization, but Long-standing Challenges Remain
paragraphs: DOD continues to take steps to comply with the requirements of the Act  and to satisfy relevant systems modernization management guidance. In  particular, DOD released an update to its corporate BEA (version 6.0) and  ETP, and issued its annual report to Congress describing steps that have  been taken and are planned relative to the Act’s requirements, among  other things. Collectively, these steps address several statutory provisions  and best practices concerning the BEA, transition plan, budgetary  disclosure, and investment review of systems costing in excess of $1  million. However, the pace of DOD’s progress in defining and  implementing these key modernization management controls has slowed  compared with the progress the department had made, and we have  reported, each of the last 4 years. As a result, challenges that we identified  last year largely remain to be addressed to fully implement the Act and  relevant guidance. Most notably, the department has yet to extend and  evolve its BEA and to provide the total federated family of DOD parent  and subsidiary architectures for the business mission area, which are  needed to comply with the Act. It also has yet to fully define IT investment  management policies and procedures at the corporate and component  levels, and the business system information used to support the  development of the transition plan and DOD’s budget requests, as well as  certification and annual reviews, is of questionable reliability. DOD  officials agree that additional steps are needed to fully implement the Act’s  requirements and related system modernization management best  practices. Further, they stated that progress over the last year has been  slowed by yet-to-be-resolved issues surrounding the Deputy CMO and  military department CMO positions.
subsections: section_title: DOD Continues to Evolve Its Corporate BEA, but a Well-Defined Plan for Federating It Has Yet to Be Developed and Progress on Component Architectures Has Been Slow
paragraphs: Among other things, the fiscal year 2005 NDAA requires DOD to develop a  BEA that would cover all defense business systems and their related  functions and activities and that would enable the entire department to (1)  comply with all federal accounting, financial management, and reporting  requirements and (2) routinely produce timely, accurate, and reliable  financial information for management purposes. The BEA should also  include policies, procedures, data standards, and system interface  requirements that are to be applied throughout the department. As such,  the Act requires an architecture that extends to all defense organizational  components. In 2006, the department adopted an incremental and  federated approach to developing such an architecture. Under this  approach, the department releases new architecture versions every year  that include a corporate BEA that is to be augmented by a coherent family  of component architectures. As we have previously reported, such an  approach is consistent with best practices and appropriate given DOD’s  scope and size.
In 2008, we reported that the then-current version of the BEA (version 5.0)  addressed, to varying degrees, missing elements, inconsistencies, and  usability issues that we previously identified, but that gaps still remained.  On March 13, 2009, DOD released BEA 6.0, which addresses some of these  gaps. For example, it begins to address information assurance by  identifying related laws, regulations, and policies. This is important  because the nature and substance of institutionalized security  requirements, controls, and standards should be captured in the  architecture products, since information assurance permeates every  aspect of an organization’s operations. In addition, the latest version of the  BEA begins to address the technical standards (e.g., W3C XML- Encryption) needed to allow business systems to work in an  expeditionary environment, which would, among other things, allow  warfighters operating in these environments to access business systems.
Version 6.0 of the BEA also addresses, to varying degrees, missing  elements, inconsistencies, and usability issues that we previously  identified, but continues to be missing important content. Examples of  these improvements and remaining issues are summarized below.
The latest version includes 35 new business rules. As we previously  reported, business rules are important because they translate business  policies and procedures into specific, unambiguous rules that govern what  can and cannot be done. As such, they facilitate consistent implementation  of laws, policies, and procedures. Examples of new business rules in the  Common Supplier Engagement business priority area are (1) an  accepting or inspecting organization must be provided on all contracts for  goods or services and (2) both a minimum and a maximum ordering limit  must be provided when the contract is an indefinite delivery/indefinite  quantity contract. In addition to adding business rules, Version 6.0  reflects the deletion of 22 business rules that, according to DOD, were no  longer applicable and were thus obsolete. Notwithstanding these additions  and deletions, BEA 6.0 still does not provide business rules for all business  processes. For example, there are no business rules for the File  Discrepancy Report for Other Goods and Services business process in the  Common Supplier Engagement and Materiel Visibility business priority  areas. Such limitations in DOD’s business rules limit the department’s  ability to ensure that business operations and supporting systems are  properly implemented.
The latest version includes additional information on important security  architecture content. For example, it now identifies information assurance  laws, regulations, and policies and describes information assurance  characteristics of key information exchanges (e.g., Awarded Contract is  designated as a sensitive information exchange). However, not all  financial information exchanges (e.g., Receipt Account Trial Balance and  Ledgers) include such key information assurance characteristics as  confidentiality, integrity, and nonrepudiation. Without specifying such  information assurance characteristics for all relevant exchanges, DOD will  be limited in its ability to implement adequate security controls into the  systems that support these exchanges.
The latest version continues to add new operational activities, which  describe actions performed in conducting DOD business operations (e.g.,  Deliver Property and Forces). These operational activities are important  because they are DOD’s primary basis for determining if a system is being  defined in a way that is compliant with the BEA. However, key operational  activities are not yet included in the BEA. For example, Version 6.0 still  does not include the Foreign Military Sales operational activity, which  affects multiple DOD business missions and organizations. Without  including such important operational activities, programs do not have all  the information necessary for determining if they are compliant with  applicable constraints (e.g., data definitions and business rules).
The latest version includes updates on the information that flows among  operational nodes (i.e., organizations, business operations, and system  elements). Information flows are important because they define what  information is needed and where and how the information moves to and  from operational entities. While Version 6.0 adds approximately 50 new  information exchanges (e.g., Approved Payment Request) among  business functions and approximately 15 data exchanges (e.g., Payment  Request for Goods) among system functions, it still contains information  exchanges (e.g., Accounts Payable Account) that are not attached or  linked to any operational nodes (or organizations). Further, this version’s  information-related architecture products contain inconsistencies. For  example, information exchanges such as Final Contract or Order Costs  and Estimate at Completion are listed in the information exchange  integrated dictionary, but are not listed in the operational information  exchange product. As a result, DOD’s ability to understand how  information is shared among operational entities, and subsequently  develop or modernize systems that can effectively share such information,  will be constrained.
The latest version also depicts end-to-end business flows (e.g., Budget to  Report) with linkages to BEA business processes (e.g., Execute  Apportionment and Allocate Funds). However, BEA 6.0 does not include,  for each end-to-end business flow, a create, read, update, and delete  matrix that shows how the business processes and their associated  applications manage specific data objects (e.g., Approved  Apportionment). These matrices are important because they reveal  natural groupings of business activities and data objects, and thus are used  to identify business activities to be automated. Without this information,  DOD will be limited in its ability to develop a target architecture that  effectively integrates information and systems that support its business  activities.
BTA officials recognize many of these issues and state that they will be  addressed as the BEA continues to evolve. In this regard, the Chief  Architect stated that the process for evolving the BEA is described in the  architecture’s Concept of Operations. Specifically, it describes a process  that calls for business cases to justify proposed improvements that are  then prioritized and used to create a BEA plan for DBSMC approval.  However, the Concept of Operations has yet to be approved, and available  documentation does not demonstrate that this process is being followed.  Further, we have yet to receive an architecture plan or evidence of  DBSMC approval of such a plan. As we have previously reported and  recommended, BTA needs an enterprise architecture program  management plan that defines what the department’s incremental  improvements to the architecture (and transition plan) will be, and how  and when they will be accomplished, including what (and when)  architecture and transition plan scope and content and architecture  compliance criteria will be added into which versions. BTA has not yet  developed such a plan. According to BTA officials, the department’s next  steps are contingent upon ongoing discussions about how architecture  planning will be affected by the Deputy CMO’s efforts to align the  department’s various planning activities in its strategic management plan,  which is to be issued no later than July 1, 2009. These discussions will be  further complicated by the lack of clarity surrounding the Deputy CMO’s  roles, responsibilities, and authorities and how the deputy will work with  other senior leaders across the department who have responsibility for  business operations.
Beyond the above-discussed limitations, Version 6.0 also continues to  represent only the thin layer of corporate architectural policies,  capabilities, rules, and standards that apply DOD-wide (i.e., to all DOD  federation members). This means that Version 6.0 appropriately focuses  on addressing a limited set of enterprise-level (DOD-wide) priorities and  providing the overarching and common architectural context that the  distinct and substantially autonomous member (i.e., component)  architectures inherit. However, this also means that Version 6.0 does not  provide the total federated family of DOD parent and subsidiary  architectures for the business mission area.
subsections: section_title: DOD’s Progress in Federating Its BEA Has Been Slow
paragraphs: Recognizing the need to address its component architecture challenges,  BTA released an update to its initial business mission area federation  strategy and road map in January 2008. Among other things, this strategy  was to address how the corporate BEA would be extended to the military  departments and defense agencies and how business services will be  identified and delivered across the business mission area. (See fig. 3 for a  conceptual representation of DOD’s federated BEA.)
In September 2006, DOD issued its initial business mission area federated  strategy and road map, which we reported lacked adequately defined tasks  needed to achieve the strategy’s goals, such as addressing how strategy  execution will be governed, component architectures will be aligned with  the latest version of the BEA, and common applications and systems  across the department will be identified and reused. Accordingly, we  reiterated our prior recommendation for a BEA management plan, and  recommended that DOD ensure that this plan describes, at a minimum,  how the business mission area architecture federation would be governed;  how the business mission area federation strategy alignment with the DOD  architecture federation strategy would be achieved; how component  business architectures’ alignment with incremental versions of the BEA  would be achieved; how shared services would be identified, exposed, and  subscribed to; and what milestones would be used to measure progress  and results.
In January 2008, DOD issued an updated strategy, and in May 2008, we  reported that the update, along with the associated global information grid  strategy, partially addressed our recommendations. Specifically, we  reported that the strategies provided high-level roles and responsibilities  for federating the architecture and additional definition around the tasks  needed to achieve alignment among DOD and component architectures.  We also noted that the strategy for the business mission area provided for  conducting pilot programs across the components to demonstrate the  technical feasibility of architecture federation, and for using the lessons  learned from the pilots to improve and update the strategies.
To their credit, BTA and other DOD entities, such as ASD(NII)/DOD CIO  and the Department of the Army, are collaboratively taking steps to  establish the foundation for implementing the strategy. For example, they  have    selected the Department of the Army’s Defense Knowledge Online to be  BTA’s federated enterprise portal, which is to be the point of access to  information about all DOD and component architectures, and is to allow  users to search and navigate through this information;   established and are using the DOD Architecture Registry System, which is  maintained by ASD(NII)/DOD CIO, as the repository to contain  architecture content;   conducted five pilots at three military departments and two defense  agencies to evaluate various aspects of architecture federation and  develop lessons learned about, for example, approaches for capturing and  managing architecture metadata (Air Force pilot), and enterprise search  and discovery methods (Navy pilot); and   developed guidance on identifying and registering business services and,  as of November 2008, identified and registered 25 business services, such  as a service that provides detailed information on each aircraft at the base  (e.g., an aircraft’s mission capability and maintenance status), and a  service that allows aircraft maintenance data to be retrieved, created,  updated, and removed.
According to officials from ASD(NII)/DOD CIO, which is responsible for  overall DOD architecture federation, the results of the pilots are being  used to determine future federation steps for all DOD mission areas. In  addition, BTA officials said that both BTA and ASD(NII)/DOD CIO are  defining a basic set of standard architecture models, including a common  vocabulary for using architecture information across DOD, to allow for  uniform representation of architecture content. Establishing such a  common framework is important because DOD’s current lack of uniform  representation for enterprise architecture content, according to BTA and  ASD(NII)/DOD CIO officials, will limit the understanding and utility of the  federated architecture.
Notwithstanding the above steps, BTA’s strategy for federating the BEA  still does not contain sufficient detail to permit effective and efficient  execution and adequately address our recommendations. For example, the  business mission area’s federation implementation road map only outlines  high-level, near-term milestones, such as milestones for developing a  governance charter for the DOD CIO Enterprise Guidance Board, which is  DOD’s senior forum for guiding the development and approval of  enterprise-level guidance (including IT policy, architecture, and standards)  on enterprise architecture, and conducting a pilot with Defense  Knowledge Online to test an access control mechanism. It does not, for  example, specify tasks to be performed to achieve those milestones,  identify milestones or tasks beyond fiscal year 2010, or identify resources  needed to perform tasks (e.g., funding, staffing, tools, and training).  Further, the strategy does not describe how the various architecture  federation activities taking place across DOD come together over time to  achieve a federated BEA, including measurement of progress, results, and  the component architectures’ alignment with the latest version of the BEA.  BTA and ASD(NII)/DOD CIO officials stated that these details have yet to  be described because of unresolved issues surrounding the Deputy CMO  and military department CMO positions.
Moreover, DOD’s federation efforts have yet to benefit from any  independent verification and validation (IV&V) assessments. As we  previously reported, such assessments are important to ensure the  completeness, consistency, understandability, and usability of the  federated family of architectures. Accordingly, we recommended that  DOD have its BEA IV&V contractor perform such assessments and  disclose the results in its annual report to Congress. However, DOD’s  March 2009 annual report does not include this information. According to  BTA officials, from October 2007 through March 2009, BTA expended  approximately $3 million on BEA-related IV&V activities. However, these  activities have focused on the corporate BEA and not the entire federated  family of architectures. BTA officials also stated that future IV&V activities  are not currently focused on the federated family of architectures. They  added that they are engaged in discussions with ASD(NII)/DOD CIO on  how and who to best perform such assessments, given that the federated  BEA is a part of DOD’s overall federated enterprise architecture, which is  led by ASD(NII)/DOD CIO.
The challenges that the department faces in federating its BEA, and the  importance of disclosing to congressional defense committees the state of  its federation efforts, are amplified by the current state of the military  departments’ enterprise architecture programs. Specifically, we recently  reported that none of the three military departments could demonstrate  through verifiable documentation that it had established all of the core  foundational commitments and capabilities needed to effectively manage  the development, maintenance, and implementation of an architecture,  which are outlined in our Enterprise Architecture Management Maturity  Framework. While the Air Force’s architecture efforts are well ahead of  those of the Navy and Army, all three had yet to fully satisfy important  aspects of our framework. Examples of their architecture limitations are  discussed below:    None of the military departments had fully defined its “as is” and “to be”  architecture environments and associated transition plans. This is  important because without a full understanding of architecture-based  capability gaps, the departments would not have an adequate basis for  defining and sequencing their ongoing and planned business system  investments.
None of the military departments had fully addressed security as part of its  respective “as is” and “to be” environments. This is important because  security is relevant and essential to every aspect of an organization’s  operations, and therefore the nature and substance of institutionalized  security requirements, controls, and standards should be embedded  throughout the architecture, and reflected in each system investment.
None of the military departments was using an IV&V agent to help ensure  the quality of its architecture products. IV&V is a proven means for  obtaining unbiased insight into such essential architecture qualities as  completeness, understandability, usability, and consistency.
None of the military departments could demonstrate that its IT  investments were actually in compliance with its architecture. This is  relevant because the benefits from using an architecture, such as  improved information sharing, increased consolidation, enhanced  productivity, and lower costs, cannot be fully realized unless individual  investments are actually in compliance with, among other things,  architectural rules and standards.
To address these limitations, we made recommendations aimed at  improving the management and content of these architectures. DOD  agreed with our recommendations. However, our recommendations have  yet to be fully implemented. Specifically, none of the military departments  provided documentation demonstrating that the above-cited limitations  have been addressed. Until DOD has a well-defined family of architectures  for its business mission area, it will not fully implement the requirements  of the Act and will remain challenged in its ability to effectively manage its  business system modernization efforts.
section_title: DOD Continues to Update Its ETP, but Important Elements Are Still Missing, as Are Individual Component Plans
paragraphs: Among other things, the Act requires DOD to develop an ETP for  implementing its BEA that includes listings of the legacy systems that will  and will not be part of the target business systems environment and  specific time-phased milestones and performance metrics for each  business system investment.
On September 30, 2008, DOD released the latest version of its ETP,  which  in general provides information on about 645 business systems, including,  to varying degrees, the required information on 54 systems that are linked  to key transformational objectives and priorities. For example, it  includes specific time-phased milestones with status indicators (e.g., met,  on track, or deleted) for about 47 out of the 54 systems, and it includes  performance metrics (e.g., voucher payment time and integration test  progress) for about 26 of these. Further, the latest version of the E discusses progress made since March 2008 on business system  investments, as well as descriptions of planned near-term activities (e. g.,  next 12 months). However, previously identified limitations in the scope  and completeness of the latest version of the ETP remain. Examples of  improvements and remaining issues are su mmarized below.
The ETP provides a range of information for some, but not all, business  system investments, such as 3 years of budget information for about 342  out of 645 systems (about 50 percent), 46 of which are linked to key  transformation objectives and priorities. However, the ETP does not yet  include system and budget information for all the business systems  identified in the department’s IT systems repository. According to the ETP,  it does not include budget information for about half of the business  systems identified because the budget data for some of these systems  were not included in the fiscal year 2009 budget submission. Further,  according to BTA officials, the ETP continues to focus on tier 1 and 2  business systems. However, not all DOD components have developed  subordinate transition plans that would address all the business system  investments. For example, as we reported last year, the Navy and Army  have not yet developed subordinate transition plans. More specifically,  Navy officials stated that they are revising their enterprise architecture  development and governance approach and, according to draft Navy  enterprise architecture documentation associated with this approach, an  enterprise architecture transition plan will be developed. Further, as we  reported, the Air Force’s transition plan is limited. For example, it is not  based on an analysis of the gap in capabilities between the department’s  “as is” and “to be” environments. Collectively, this means that a complete  family of DOD and component transition plans does not exist. According  to the BTA official responsible for the ETP, BTA and the military  departments are currently discussing whether component-level plans  should be published separately from or incorporated into the corporate  ETP. This is further complicated by the uncertainty surrounding how the  Deputy CMO will work with other senior leaders who have responsibility  for business operations, including the military department CMOs.
The ETP continues to provide performance measures for some, but not all,  enterprise and component investments (i.e., programs), including key  milestones (e.g., initial operating capability) and status indicators.  However, the plan has yet to include other important information needed  to understand the sequencing of new systems becoming operational and  legacy systems being phased out. In particular, the planned investments  have not been sequenced based on a range of important factors cited in  federal guidance, such as technology opportunities, marketplace trends,  fiscal and budgetary constraints, institutional system development and  acquisition capabilities, new and legacy system dependencies and life  expectancies, and the projected value of competing investments. Rather,  the ETP continues to be largely based on a bottom-up process in which  ongoing programs have been compiled and categorized in the plan around  business enterprise priorities. For example, many of these investments are  dependent on Net-Centric Enterprise Services, and as such the plans and  milestones for each should reflect the incremental capability deployment  of these enterprise services.
The ETP and the business mission area federation strategy describe the  department’s approach to enterprise application integration, including  plans for using specific services and standards for integrating financial  application systems. Including such information in the ETP and  associated documentation will help to clarify relationships and  dependencies among legacy applications and systems and new or  modernized applications and systems. However, all systems needed to  achieve integration are not specified. For example, the ETP does not  identify all of the systems that must be integrated for each end-to-end  business flow (e.g., budget-to-report) to support activities that are cross- functional and cross-cutting across organizational boundaries.
The ETP does not include all legacy systems that will not be part of the  target BEA and does not provide the schedule for terminating these legacy  systems, as required by the Act. For example, while the Navy Enterprise  Resource Planning program’s August 2008 investment review board  documentation identifies 41 legacy systems, the ETP identifies only 25 of  these systems. In addition, the plan is missing information about some  legacy systems and modernization programs. Specifically, the plan does  not include termination dates for 40 out of 514 legacy systems. Including a  comprehensive and reliable list of legacy systems is important for the  department to have a meaningful and reliable basis for managing the  disposition of legacy systems and for sequencing the introduction of  modernized business operations and supporting systems.
BTA officials said that a number of actions are envisioned to address the  above-cited areas and further improve the ETP, such as working with the  military departments and defense agencies to determine which systems  should be included in the corporate-level ETP and ensuring that the next  version of the ETP includes more information about dependencies among  systems. Until the ETP, or a federated family of such plans, either directly  or by reference includes relevant information on the full inventory of  investments across the department (and does so in a manner that reflects  consideration of the range of variables associated with a well-defined  transition plan, such as timing dependencies among investments and the  department’s capability to manage them), it will not provide a sufficient  basis for sequencing the introduction of modernized systems. To help  DOD improve its ETP, we have previously made recommendations that  the department is in the process of addressing aimed at formalizing its  plans for incrementally improving its transition plan.
section_title: Fiscal Year 2009 Budget Submission Did Not Include Key Information on All Business Systems
paragraphs: Another requirement of the Act is that DOD’s annual IT budget submission  must include key information on each business system for which funding  is being requested, such as the system’s designated approval authority and  the appropriation type and amount of funds associated with  development/modernization and current services (i.e., operation and  maintenance).
As we reported last year, the department’s fiscal year 2009 budget  submission included a range of information required by the Act on  business system investments. Specifically, for 273 investments that  involve development/modernization activities, the submission included  such information as the system’s (1) name, (2) approval authority, and (3)  appropriation type. The submission also identified the amount of the fiscal  year 2009 request that was for development/modernization versus  operations/maintenance. Further, for those system investments in excess  of $1 million in modernization funding, the submission cited the  certification status (e.g., approved, approved with conditions, not  applicable, and withdrawing) and the DBSMC approval date, where  applicable.
However, the fiscal year 2009 budget submission does not reflect all  business system investments. To prepare the submission, DOD relied on  business system investment information (e.g., funds requested, mission  area, and system description) that is entered by the components into  DOD’s Select and Native Programming Data Input System–Information  Technology (SNAP-IT). In accordance with DOD guidance and according  to ASD(NII)/DOD CIO officials, the business systems listed in SNAP-IT  should match the systems listed in the Defense Information Technology  Portfolio Repository (DITPR)—the department’s authoritative business  systems inventory. However, the number of business systems in DITPR is  unclear. Specifically, in March 2009, DITPR data provided by DOD  included about 6,800 systems, and in April 2009, BTA officials stated that  the number of operational business systems in the repository was 2,480,  adding that the 6,800 number included systems that were not business  systems and systems that may no longer be operational. However, they  have yet to provide support for this revised number of business systems.
Regardless, SNAP-IT is potentially missing thousands of business systems  that are identified in DITPR. Specifically, SNAP-IT contains about 2,100  systems, of which only about 1,500 are categorized as business systems.  Restated, the fiscal year 2009 budget submission is missing somewhere  between 980 and 5,300 business systems. For example, the Department of  the Navy’s Personnel Information System for Training, Operations, and  Logistics and the Air Force’s Contractor Responsibility Information  System are listed in DITPR but not listed in SNAP-IT. Moreover, as stated  earlier in the report, DOD has also recognized limitations in its budget  submission in its ETP. The ASD(NII)/DOD CIO official responsible for  administering the SNAP-IT data said that while the components are  responsible for ensuring that information about their respective systems is  accurate and complete, the department recognizes the need to reconcile  the information between SNAP-IT and DITPR to improve the systems’  comprehensiveness and accuracy. However, the department has yet to  develop a plan or time frame for doing so. Without a reliable  comprehensive inventory of all defense business systems, DOD will not be  able to ensure the completeness and reliability of its IT budget  submissions.
section_title: DOD Has Made Progress in Establishing Corporate and Component Investment Management Structures, but Associated Policies and Procedures Are Not Yet Fully Defined and Implemented
paragraphs: The Act also requires DOD to establish business system investment review  structures, such as the previously discussed DBSMC and five IRBs, as well  as processes that are consistent with the investment management  provisions of the Clinger-Cohen Act. As we have previously reported,  organizations that satisfy Stages 2 and 3 of our ITIM framework have the  investment selection, control, and evaluation structures, and the related  policies, procedures, and practices that are consistent with the investment  management provisions of the Clinger-Cohen Act.
DOD and the Air Force have largely established the kind of investment  management structures provided for in the Act and our ITIM framework.   However, the Navy has not. Moreover, neither DOD nor these components  have defined the full range of related investment management policies and  procedures that our framework identifies as necessary to effectively  manage investments as individual business system programs (Stage 2) and  as portfolios of programs (Stage 3). Until all of DOD has put these  requisite investment management structures and supporting policies and  procedures into place, the billions of dollars that the department and its  components invest annually in business systems will remain at risk.
subsections: section_title: Corporate and Air Force Investment Management Structures Are Largely Established, but Navy Structures Remain a Work in Progress
paragraphs: DOD has largely established corporate-level organizational structures that  are associated with Stages 2 and 3 of our framework. As we reported in  May 2008, the department has an enterprisewide investment board and  four subordinate boards, and has assigned them responsibility for business  systems investment governance, including conducting investment  certification and approval reviews and annual reviews as provided for in  the Act. The enterprisewide board—the DBSMC—is composed of the  department’s top executives, such as the Deputy Secretary of Defense and  the ASD(NII)/DOD CIO, as provided for in the Act. Among other things,  the DBSMC is responsible for establishing and implementing policies  governing the organization’s investment process and approving lower-level  investment board processes and procedures. The subordinate boards  include four IRBs that are composed of senior officials representing their  respective business areas, including representatives from the combatant  commands, defense agencies, military departments, and Joint Chiefs of  Staff. Among other things, the IRBs are responsible and accountable for  overseeing and controlling certain business system investments, including  ensuring compliance and consistency with the BEA. The department has  also assigned responsibility to the Under Secretary of Defense for  Acquisition, Technology, and Logistics for managing business system  portfolio selection criteria.
Since 2008, the department has taken additional steps to establish a fifth  IRB, the DOD Chief Information Officer’s review board, which is to  oversee investments in business systems whose primary purpose is to  support infrastructure and information assurance activities. According to  DOD officials, this board is to replace the Enterprise Information  Environment Mission Area review board, which was the fifth board  required by the Act, and its charter has been drafted, but not approved.
With respect to the military departments’ investment management  structures, we reported in May 2008 that the Air Force had established the  organizational structures associated with Stages 2 and 3 of our framework,  such as a business systems IRB consisting of senior executives from the  functional business units, including the Office of the Air Force CIO.  Among other things, this board is responsible for business system  investment governance, including conducting investment precertification,  approval, and annual reviews, as required by the Act.
We recently reported that, in contrast to the Air Force, the Navy had not  yet established an enterprisewide IRB composed of senior executives from  its IT and business units, to define and implement a Navy-wide business  system governance process. We concluded that without such structures,  the Navy’s ability to ensure that business system investment decisions are  made consistently and reflect the needs of the organization was limited.  Accordingly, we recommended that the Navy establish these management  structures. Navy officials told us that a Secretary of the Navy Instruction  that is intended to address these limitations has been drafted but not yet  approved.
section_title: Corporate and Air Force Investment Management Policies and Procedures Are Being Established, but Navy’s Remain Largely Undefined
paragraphs: DOD has partially defined the full range of corporate and component-level  policies and procedures that we previously recommended it establish to  effectively support project-level (Stage 2) and portfolio-based (Stage 3)  investment management practices. Specifically, DOD recently issued new  corporate-level policies and procedures that further address key practices  in our ITIM framework associated with project-level investment  management (Stage 2), such as instituting the investment board and  providing investment oversight. In particular, DOD’s revised 2008  acquisition policy and draft business capability life cycle acquisition  policy and guidance outline aspects of how the business investment  review processes are to be coordinated with other decision-support  processes used at DOD, such as the Joint Capabilities Integration and  Development System and the Defense Acquisition System. For example,  the revised policies and guidance now require a team to assess the risks  associated with each Major Automated Information System and to share  the results with the program manager and component functional  sponsor, who in turn are to collaboratively report the risks to both the  IRB and the program’s milestone decision authority prior to each  milestone decision. They further require the DBSMC to approve the  obligation of funds prior to the first milestone review of each major  business system. In addition, DOD also recently revised its policy for  overseeing the acquisition of systems that provide joint capabilities, to  require all business system investments to comply with the business  system investment review process and the business capability life cycle  process.
The department has also recently established guidance associated with portfolio-level investment management (Stage 3) practices. However,  DOD’s updated corporate-level policies and procedures are still missing  critical project- and portfolio-based investment man we previously recommended, as discussed below.
Policies and procedures for instituting the investment board do not  address how all investments that are past the development/modernization  stage (i.e., in operations and maintenance) are to be governed. Given that  DOD invests billions of dollars annually in operating and maintaining  business systems, this is significant. For example, while the 2009 up date to  the IRB guidance now requires an annual review of all investme previously certified by IRBs, including those in operations and  maintenance, this review is not required for systems in operations and maintenance that were not previously certified by the IRBs. Our ITIMframework emphasizes that the corporate investment boards should  review important information about an investment, such as cost and  performance baselines, throughout the investment’s life cycle. In addition,  while the department’s investment process addresses how investme nt- related processes are to be coordinated with the Joint Capabilities  Integration and Development System and the Defense Acquisition System,  these processes do not apply to all business systems. For example, DO D’s  updated acquisition policy states that IRB involvement in acquisition  decisions is required only for Major Automated Information Systems.  Moreover, the 2008 acquisition policy and draft business capability life  cycle acquisition policy and guidance do not address how these process are to be coordinated with the Planning, Programming, Budgeting, an Execution process. Furthermore, the business capability life cycle  acquisition policy and guidance has yet to be approved. Without approved policies and procedures that provide clear visibility into all inves including linkages to related mana gement systems, inconsistent  investment decisions may result.
Procedures for selecting an investment do not specify how the IRBs use  the full range of cost, schedule, and benefit data in making selection (i.e.,  certification) decisions. Specifically, while the revised 2009 IRB guida nce  states that the IRBs will consider cost, schedule, and benefit data in  making certification decisions, the guidance does not define how the  boards are to consider these factors. According to our ITIM guidance, a  structured selection method should provide investment boards, business units, and IT developers with a common understanding of the selection  process to be followed, including how cost, schedule, and benefit data are  to be used to compare and select projects. Furthermore, while DOD issued  an IRB roles and responsibilities policy in January 2009 that states that the  c ertification authorities will define the selection criteria for determining   whether an investment is to be an enterprisewide system or remain  component specific, those certification authorities have yet to do so.  Without documenting how the IRBs employ such factors when making  selection decisions, the department cannot ensure that the boards  consistently and objectively select proposals that best meet the  department’s needs and priorities.
Policies and procedures for overseeing an investment do not provide for  sufficient visibility into component-level investment management  activities, including component reviews of systems in operations and  maintenance and smaller investments, commonly referred to as tier 4  investments. Such visibility is important because DOD reports that only  346 system modernization efforts have been IRB certified and DBSMC  approved. This means that the vast majority of business systems are  reviewed and approved only within the component organizations. While  the January 2009 IRB roles and responsibilities policy requires that each  component submit an end-of-the-fiscal-year report listing those systems  that have been reviewed by the cognizant IRB, this report lacks important  project information. For example, it does not address components’  adherence to cost, schedule, and risk investment selection and control  criteria. According to our ITIM framework, an investment board should  have visibility into each project’s performance and progress toward  predefined cost, schedule, and benefit expectations as well as each  project’s exposure to risk. Without such visibility, DOD components risk  making investment decisions that are inconsistent and not fully grounded  in objective data.
Policies and procedures have not been fully established for defining the  portfolio selection criteria or for creating and evaluating the portfolio of  business systems. Specifically, the department has assigned responsibility  to its certification authorities for defining the criteria to be used for  making portfolio selection decisions, creating portfolios, and evaluating  the performance of portfolio investments. However, these authorities have  yet to fulfill these responsibilities.
According to our ITIM framework, the development and use of portfolio  selection criteria focuses on the synergistic benefits to be found among an  agency’s entire collection of investments, rather than just from the sum of  the individual investments.
Policies and procedures for conducting postimplementation reviews do  not address all business systems. Specifically, in its January 2009 update to  its IRB guidance, the department added a new type of review, called a  closeout annual review, to be performed when a business system  modernization has been completed. According to the guidance, this review  is to function as a postimplementation review for IRB-certified systems  and is to provide the IRBs with lessons learned and metrics about  completed investment efforts. However, the guidance does not address  how expected benefits were achieved. According to our ITIM framework,  examining the differences between estimated and actual investment costs  and benefits is a key aspect of conducting postimplementation reviews.
According to BTA officials, these limitations are due to the newness of its  investment review policies and procedures, which they said will be revised  over time to address the limitations. Adequately documenting both the  policies and the associated procedures that provide predictable,  repeatable, and reliable investment selection and that control and govern  how an organization manages its IT investment portfolios reduces  investment risk of failure and provides the basis for rigor, discipline, and  repeatability in how investments are selected and controlled across the  entire organization.
With respect to the military departments’ investment management policies  and procedures, we recently reported that the Air Force and the Navy did  not have fully documented policies and procedures for overseeing the  management of business system investments and for developing and  managing complete business systems investment portfolios. To address  these areas, we made recommendations aimed at implementing our  framework’s Stage 2 and 3 practices, and DOD partially agreed with these  recommendations. Under DOD’s tiered accountability approach to  reviewing and approving business systems investments, in which  investment review begins at the component level and proceeds through a  hierarchy of review and approval authorities, depending on the size and  significance of the investment, it is vital that DOD components implement  these practices. BTA officials told us that the success of the department’s  overall process for managing business system investments depends on  each component performing a thorough analysis and making informed  decisions relative to each business system before it is submitted for  higher-level review and approval.
To the Air Force’s credit, it has recently updated its policies and  procedures to address our project-level investment management  recommendations (Stage 2 of our framework). For example, the Air  Force’s recently developed IT investment review guidance provides for the  review of all business systems, to include those in operations and  maintenance, and it defines the process by which its IRB will review these  systems. Further, the guidance specifies how business investments,  including those in operations and maintenance, are to be prioritized using  factors such as mission and strategic value and risk. The Air Force has  also addressed key practices associated with portfolio-level investment  management (Stage 3), such as creating and modifying IT portfolio  selection criteria and assigning responsibility for the development and  modification of IT portfolio selection criteria. Specifically, the guidance  describes the criteria to be used to make portfolio selection, assigns  responsibility for developing the criteria to an integrated working team,  and assigns responsibility for approval of the criteria to a senior working  group.
However, the Air Force’s recent investment review guidance is still  missing critical elements needed to effectively carry out essential  investment management activities. For example, the guidance does not yet  specify how the business investment management activities are  coordinated with other DOD management systems, such as the Joint  Capabilities Integration and Development System, the Defense Acquisition  System, and the Planning, Programming, Budgeting, and Execution  process. Further, the guidance does not provide for sufficient oversight  and visibility into investment management activities. Specifically, while  the Air Force has predefined criteria for adherence to cost, schedule, and  performance milestones, and requires the development of corrective  actions when a system deviates from milestones, it does not have policies  and procedures that guide the implementation of these corrective actions  when program expectations are not met. Moreover, the Air Force has yet  to develop policies and procedures for maintaining investment portfolios.  According to the Air Force, such key practices will be addressed in future  revisions to its guidance.
In contrast, the Navy has not made as much progress as the Air Force in  addressing either our project-level or portfolio-level recommendations.  For example, the Navy has yet to fully document policies and procedures  for overseeing the management of business system investments and for  developing and managing complete business systems investment  portfolios. Among other things, it does not have policies and procedures  that specify decision-making processes for program oversight and describe  how corrective actions should be taken when projects deviate from their  project management plans. According to the Navy, a policy for addressing  our recommendations has been drafted, but has yet to be approved.
As discussed in our ITIM framework, adequately documenting both the  policies and associated procedures that govern how an organization  manages its IT projects and investment portfolios is important because  doing so provides the basis for rigor, discipline, and repeatability in how  investments are selected and controlled across the entire organization.  Until these missing policies and procedures are fully defined at both the  corporate and the component levels, it is unlikely that the thousands of  DOD business system investments will be managed in a consistent,  repeatable, and effective manner.
section_title: DOD Continues to Certify and Approve Business Systems, but Decisions Are Sometimes Based on Limited Information
paragraphs: The Act specifies two basic requirements that took effect October 1, 2005,  relative to DOD’s use of funds for business system modernization  investments that involve more than $1 million in obligations. First, it  requires that these investments be certified by a designated approval  authority as meeting specific criteria, such as   demonstrating compliance with the BEA. Second, it requires that the  DBSMC approve each of these certifications, adding that failure to do so  before the obligation of funds for any such investment constitutes a  violation of the Anti-deficiency Act. In addition, DOD’s business system  approval and certification guidance directs programs to submit additional  information, such as a program’s economic analysis, to designated  approval authorities.
As it has since 2005, DOD continues to certify and approve business  system modernization investments in excess of $1 million. However, since  2006, we have identified limitations in the information used to certify and  to approve several major programs. Moreover, although IRB certification  and annual review guidance calls for DOD’s authoritative business systems  repository (i.e., DITPR) to be used to inform business system investment  certification and annual review decisions, information in this repository is  not always current and accurate. As a result, DOD risks making  certification and approval decisions that are not prudent and justified.
subsections: section_title: DOD Has Continued to Certify and Approve Business Modernizations in Excess of $1 Million
paragraphs: The department has established an approach to meeting the Act’s  requirements that reflects its philosophy of tiered accountability. Under  this approach, investment review begins within the military departments  and defense agencies and advances through a hierarchy of review and  decision-making authorities, depending on the size, nature, and  significance of the investment. For those investments that meet the Act’s  dollar thresholds, this sequence of review and decision making includes  component precertification, IRB certification, and DBMSC approval. For  those investments that do not, investment decision-making authority  remains with the component. This review and decision-making approach  has two types of reviews for business systems: certification/approval  reviews and annual reviews.
Certification/approval reviews. Certification/approval reviews apply to  new modernization investments with planned obligations in excess of $1  million. These reviews focus on program alignment with the BEA and  must be completed before components obligate modernization funds. Tier  1, 2, and 3 investments that involve development and modernization funds  are certified and approved at three levels—component precertification,  IRB certification, and DBSMC approval.
At the component level, program managers are responsible for the  information about their respective programs that is in DITPR. Examples of  information contained in DITPR are regulatory compliance reports,  architectural profiles, financial benefit information (i.e., benefit-to-cost  ratio), and system life cycle costs. According to the process, the  component precertification authority is responsible for precertifying BEA  compliance and reviewing system modernization funding requests, in  addition to ensuring that IRBs receive complete, current, and accurate  information within the prescribed deadlines. The precertification authority  asserts the status and validity of the investment information by submitting  a component precertification letter to the appropriate IRB.
At the corporate level, the IRB reviews the precertification letter and  related material, and if it decides to certify the investment, prepares a  certification memorandum for the designated certification authority’s  signature that documents the IRB’s decisions and any related conditions.  The memorandum is forwarded to the DBSMC, which either approves or  disapproves the IRB’s decisions and issues a memorandum containing its  decisions. If the DBSMC disapproves a system investment’s certification, it  is up to the component precertification authority to decide whether to  resubmit the investment after it has resolved the relevant issues.
Annual reviews. The annual reviews apply to all business system  investments and are intended to determine whether the investment is  continuing to comply with the BEA, meeting its milestones, and  addressing its IRB certification conditions. Tier 1, 2, and 3 business system  investments are annually reviewed by the relevant component and IRB.
At the component level, program managers update information on all tiers  of system investments that are identified in their component’s data  repository. For Tier 1, 2, or 3 systems that are in development or being  modernized, information is updated on cost, milestones, and risk  variances and actions or issues related to certification conditions. The  component precertification authority then verifies and submits the  information for these investments to the appropriate IRB in an annual  memo.
At the IRB level, Tier 1, 2, and 3 business system development or  modernization investment reviews focus on program compliance with the  BEA, program cost and performance milestones, and progress in meeting  certification conditions. IRBs can advise the DBSMC to revoke a  certification when the investment has significantly failed to achieve  performance commitments (i.e., capabilities, schedule, and costs). When  this occurs, the component must address the IRB’s concerns and resubmit  the investment for certification.
Since October 1, 2005 (the effective date of the relevant provision of the  Act), DOD has continued to certify and approve investments with  obligations in excess of $1 million. Since fiscal year 2005, DOD has  reported that the DBSMC had approved system modernization efforts for a  total of 346 systems. According to DOD:    All but one of the 346 system modernization efforts were certified and  approved as meeting the first condition in the Act—being in compliance  with the BEA. These systems involved about $8.5 billion in  development/modernization funding.
About 60 percent of the 346 system modernization efforts (208) are owned  by the military departments and were accordingly precertified within the  military departments. More specifically, 63 were precertified within the Air  Force, 79 within the Army, and 66 within the Navy.
section_title: DOD Certification and Approval Decisions Have Been Based on Limited Information
paragraphs: Although DOD has been meeting the Act’s requirement to certify and  approve business system modernization programs, it has at times relied on  limited information in doing so. For example, we recently reported that  two large Navy business system programs did not adequately demonstrate  compliance with the department’s federated BEA, even though each  program largely followed DOD’s existing compliance guidance, used its  compliance assessment tool, and was certified and approved as being  compliant by department investment oversight and decision-making  entities. In particular, these programs’ BEA compliance assessments did  not (1) include all relevant architecture products, such as products that  specify the technical standards needed to promote interoperability among  related systems; (2) examine overlaps with other business systems, even  though a stated goal of the BEA is to identify duplication and thereby  promote the use of shared services; and (3) address compliance with the  Department of the Navy’s enterprise architecture, which is a major BEA  federation member. We attributed these limitations to various reasons,  including the fact that the department’s guidance did not provide for  performing these steps.
In addition, we reported that although the department’s investment  oversight and decision-making authorities certified and approved these  business system programs as compliant with the BEA, they did not  validate each program’s compliance assessment and assertions. According  to DOD officials, this was because responsibility for doing so is assigned  to DOD’s component organizations, such as the Department of the Navy,  under the department’s tiered accountability approach. However, the  Department of the Navy oversight and decision-making authorities also did  not validate the programs’ assessments and assertions. We concluded that  such architecture compliance limitations increase the risk of DOD  programs being defined and implemented in a way that does not  sufficiently ensure interoperability and avoid duplication and overlap.  Accordingly, we made a number of recommendations to address these  limitations, which the department agreed to implement.
Another example of limited information used to certify and approve  business system investments is the unreliable economic justifications for  the programs. According to relevant DOD guidance, the economic viability  of system investments is to be analyzed on the basis of reliable estimates  of costs and benefits. However, we have continued to report on limitations  in the cost/benefit analyses used to economically justify major DOD  business system investments. More recently, we reported that the Global  Combat Support System–Marine Corps cost estimate was not reliable, as it  was not based on historical data from similar programs and it did not  account for schedule risks, both of which are needed for the estimate to  be considered accurate and credible. In addition, we reported that the  Navy Enterprise Resource Planning program did not employ similar cost- estimating practices. As a result, we concluded that neither program had  a sufficient basis for deciding if it was the most cost-effective solution for  meeting mission needs, and we made recommendations to address these  weaknesses. DOD agreed with our recommendations.
section_title: Accurate Information about Modernization Investments Is Not Maintained in DOD’s IT System Repository
paragraphs: Since 2005, DITPR has been designated as the authoritative repository of  information about all DOD business systems. According to DOD’s business  system certification and annual review guidance, information in DITPR is  to be updated by component staff, validated by program managers, and  reviewed by component precertification authorities to ensure its accuracy,  and it is to be used by the IRBs and the DBSMC in making certification and  approval decisions, respectively.
The information in DITPR is not always accurate, and thus does not  always provide an adequate basis for informed decision making.  According to ASD(NII)/DOD CIO officials, information entered in DITPR  at the component level is not always reliable and validated. Our analysis of  selected business system information contained in DITPR confirmed such  inaccuracies:   At least 900 systems, such as the Contractor Performance Assessment  Reporting System and the Air Force’s Virtual Personnel Service  Center, showed life cycle phase start dates as the year 1900 or 1901.
At least 960 systems, such as the Armed Forces Health Longitudinal  Technology Application and BTA’s Wide Area Workflow System, show  a life cycle phase end date of 2099 or later.
Moreover, as stated earlier in this report, DOD provided inconsistent  information about the number of business systems contained in DITPR.  Specifically, in March 2009, DITPR data provided by DOD included about  6,800 systems, and in April 2009 BTA officials stated that the number of  operational business systems in the repository was 2,480. Thus, the  number of business systems in DITPR is also unclear.
According to ASD(NII)/DOD CIO officials, a policy is being developed to  have the DOD Inspector General periodically validate the accuracy of the  information in DITPR. Given that the information from DITPR is used to  make certification and approval decisions, serious limitations in the  accuracy of information could affect the quality of the decisions.
section_title: Conclusions
paragraphs: The pace of DOD’s progress in defining and implementing key institutional  modernization management controls has slowed relative to each of the  prior 4 years, leaving much to be accomplished. Specifically, the corporate  BEA continues to be missing important content, and it has yet to be  federated through development of aligned subordinate architectures for  each of the department’s component organizations. Further, while the  department has updated its strategy for federating the BEA, this strategy is  still missing important content and it has yet to be implemented.  Compounding this situation are recurring limitations in the scope and  completeness of the department’s enterprise transition plan, as well as the  immaturity of the military department architecture programs, including  the completeness of their own transition plans. In addition, the corporate  and the military departments’ approaches to business systems investment  management continue to lack the requisite structures and defined policies  and procedures to be considered effective investment selection, control,  and evaluation mechanisms. Finally, information used to support the  development of the transition plan and DOD’s budget requests, as well as  to inform certification and annual reviews, is of questionable reliability.  Collectively, these long-standing limitations in the department’s  institutional modernization management controls continue to put billions  of dollars spent each year on thousands of business system investments at  risk.
A well-defined federated architecture and accompanying transition plans  for the business mission area, along with well-defined investment  management policies and procedures across all levels of the department,  are critical to effectively addressing DOD’s business systems  modernization high-risk area. Relatedly, it is important for the department  to obtain independent assessments of the completeness, consistency,  understandability, and usability of the federated family of business mission  area architectures, including associated transition plans, and to share the  results of these assessments with its authorizing and appropriations  committees. Equally important is for the department to actually implement  its architecture and investment management controls in the years ahead  on each and every business system investment, and in doing so to ensure  that it has reliable information on each investment upon which to base  executive decision making.
Our previous recommendations to the department have been aimed at  accomplishing these and other important activities related to its business  systems modernization. While not a guarantee, having an architecture- centric investment management approach, combined with the actual  implementation of other key system acquisition disciplines that are  reflected in our existing recommendations, can provide a recipe for the  business systems modernization program’s removal from our high-risk list.  To the department’s credit, it has agreed with these recommendations and  committed to implementing them. Moreover, over the previous several  years, it has made important progress in doing so, as prior reports have  recognized. However, the pace of the progress has slowed over the last  year as the roles, responsibilities, authorities, and relationships among  recently established executive positions that are integral to defining and  implementing these controls are worked out. In light of this, it is essential  that the DBSMC, which is chaired by the DOD CMO, resolve these  positional matters, as doing so is on the department’s critical path for fully  establishing the full range of institutional management controls needed to  address its business systems modernization high-risk area.
section_title: Recommendations for Executive Action
paragraphs: Because we have existing recommendations that address most of the  institutional management control weaknesses discussed in this report, we  reiterate these recommendations.
In addition, to ensure that DOD continues to implement the full range of  institutional management controls needed to address its business systems  modernization high-risk area, we recommend that the Secretary of  Defense direct the Deputy Secretary of Defense, as chair of the DBSMC  and as DOD’s CMO, to resolve the issues surrounding the roles,  responsibilities, authorities, and relationships of the Deputy CMO and the  military department CMOs relative to the BEA and ETP federation and  business system investment management.
Further, to ensure that business system investment reviews and related  certification and approval decisions, as well as annual budget submissions,  are based on complete and accurate information, we recommend that the  Secretary of Defense direct the appropriate DOD organizations to develop  and implement plans for reconciling and validating the completeness and  reliability of information in its DITPR and SNAP-IT system data  repositories, and to include information on the status of these efforts in  the department’s fiscal year 2010 report in response to the Act.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Assistant  Deputy Chief Management Officer and reprinted in appendix II, the  department stated that it has made important progress over the past year  on its business system modernization, adding that this progress partly  addresses our prior recommendations. We agree that the department has  continued to make progress, and our report recognizes this. However, our  report also recognizes that the pace of this progress has slowed in relation  to prior years, and it links this slowdown to implementation of recent  management structural changes within the department, which DOD’s  comments acknowledge have had to occur simultaneously.
To facilitate implementation of these structural changes, we  recommended that DOD resolve the issues surrounding the roles,  responsibilities, authorities, and relationships of the Deputy CMO and the  military department CMOs relative to the BEA and ETP federation and  business system investment management. DOD partially agreed with this  recommendation. In particular, the department agreed that additional  clarity would be useful in defining the roles and responsibilities of these  positions and stated that it is committed to resolving this ambiguity  through formal policy in the near future. However, the department stated  that it believes that the Deputy CMO has the necessary authority, working  on behalf of the Deputy Secretary of Defense, and that the Deputy CMO  has a sufficiently close working relationship with the Deputy CMOs of the  military departments to make significant strides in the department’s  business operations improvement efforts, even in the absence of near-term  formal guidance. We do not agree. As we have previously reported, the  department has designated the role of the Deputy CMO as an advisor to  the CMO, and it has not assigned the Deputy CMO clear decision-making  authority. Further, the absence of clarity around the Deputy CMO’s role  and responsibilities, which DOD acknowledged in its comments,  combined with this absence of clear decision-making authority, directly  affects the nature of the Deputy CMO’s relationship with other senior  leaders in the department, as relationships are a function of roles,  responsibilities, and authorities. Therefore, we stand by our  recommendation.
With regard to our second recommendation, to develop and implement  plans for reconciling and validating the completeness and reliability of  information in its DITPR and SNAP-IT data repositories, and to include  information on the status of these efforts in the department’s fiscal year  2010 report in response to the Act, DOD stated that it partially agreed with  the recommendation. In particular, it agreed with the need to reconcile  information between the two repositories and stated that it has begun to  take actions to address this. For example, it stated that policy and  guidance now require the components to enter information in both DITPR  and SNAP-IT using what it described as a “one-to-one” relationship for all  defense business systems, and that the DOD CIO is working with the  components to facilitate implementation of this requirement. In addition, it  stated that the DOD CIO and Office of Program Analysis and Evaluation  are currently developing a plan to modify both DITPR and SNAP-IT to  eliminate duplicate data and integrate them.
Notwithstanding its actions aimed at reconciling DITPR and SNAP-IT data,  DOD commented that it disagreed that the data in the two repositories are  unreliable, stating that differences in the data between the two are due to  differences in the purpose of each repository, and that the data in each are  complete and accurate enough to support their purposes. In response, we  recognize that the repositories are used for different purposes. However,  DOD guidance calls for business system information in the two  repositories to be consistent and maintained at the same level of detail,  which, as we state in our report, is not occurring. In particular, the number  of business systems in DITPR and SNAP-IT is not consistent, which means  that one or both lack important information about DOD business systems.
As also stated in our report, system-specific information contained in  DITPR is not accurate. For example, at least 900 systems showed life cycle  phase start dates as the year 1900 or 1901, and at least 960 systems show a  life cycle phase end date of 2099 or later. In addition, during the course of  our review, DOD officials that we interviewed and who operate these  repositories recognized these data limitations and agreed that more  needed to be done to ensure data reliability.
DOD also provided technical comments on a draft of this report that we  have incorporated throughout the report, as appropriate.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; and the  Secretary of Defense. This report will also be available at no charge on our  Web site at http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: As agreed with defense congressional committees, our objective was to  assess the Department of Defense’s (DOD) actions to comply with the  requirements of section 2222 of Title 10, U.S. Code. To address this, we  used our last annual report under the Act as a baseline, analyzing whether  the department had taken actions to comply with five of the six  requirements in section 2222, related best practices contained in federal  guidance, and our prior recommendations that we previously identified as  not yet addressed. Generally, these five requirements are (1) development  of a business enterprise architecture (BEA), (2) development of an  enterprise transition plan (ETP) for implementing the BEA, (3) inclusion  of business systems information in DOD’s budget submission, (4)  establishment of business systems investment review processes and  structures, and (5) approval of defense business systems investments with  obligations in excess of $1 million. (See the background section of this  report for additional information on the Act’s requirements.) We did not  include the sixth requirement, on delegating the responsibility for business  systems to designated approval authorities, because our November 2005  report under the Act shows that it had been satisfied. Our methodology  relative to each of the five requirements is as follows:    To determine whether the BEA addressed the requirements specified in  the Act and related guidance, we analyzed version 6.0 of the BEA, which  was released on March 13, 2009, relative to the Act’s specific architectural  requirements and related guidance that our last annual report under the  Act identified as not being fully implemented. Specifically, we interviewed  Business Transformation Agency (BTA) officials and reviewed written  responses and related documentation on steps completed, under way, or  planned to address these weaknesses. We then reviewed architectural  artifacts in version 6.0 to validate the responses and identify any  discrepancies. Further, we analyzed BEA supporting documentation (e.g.,  BEA compare reports) to determine the number of additions, updates, and  deletions made to BEA artifacts (e.g., BEA business rules, data elements,  data objects, data entities, information exchanges, system data exchanges,  system entities, system functions, system interfaces, and technical  standards) as compared with the architectural content of version 5.0. We  also analyzed BEA supporting documentation to identify the number of  additions, updates, and deletions made to BEA artifacts (e.g., BEA  business rules, data objects, system data exchanges, system entities, and  system functions) that were specifically associated with the financial  visibility business enterprise priority area.
To evaluate progress made in federating DOD’s BEA, we reviewed DOD’s  Business Mission Area Architecture Federation Strategy and Roadmap  Version 2.4, released in January 2008, comparing the strategy and any  associated implementation plans with prior findings and recommendations  relative to the content of the strategy. We also obtained documentation  and interviewed cognizant DOD officials about efforts to establish a  federated DOD business mission area enterprise architecture. Further, we  reviewed the military departments’ responses on actions taken or planned  to address our previous recommendations on the maturity of their  respective enterprise architecture programs. In addition, we reviewed the  independent verification and validation (IV&V) contractor’s statement of  work and IV&V reports to determine whether they addressed the quality of  the department’s federated family of corporate and component  architectures, including the federated ETPs, and we interviewed the IV&V  contractor and BTA officials to determine plans for future IV&V work to  address the architectures’ quality.
To determine whether the DOD ETP addressed the requirements specified  in the Act, we reviewed the updated version of the ETP, which was  released on September 15, 2008, relative to the Act’s requirements and  related transition plan guidance that our last annual report under the Act  identified as not being fully implemented. Specifically, we interviewed  BTA officials and reviewed written responses and related documentation  on steps completed, under way, or planned to address these weaknesses.  We then reviewed the plan to validate the responses and identify any  discrepancies. In addition, to determine the extent to which the ETP  included system and budget information for all the business systems  identified in the department’s information technology (IT) systems  repository, we reviewed and compared the number of defense business  systems listed in the department’s authoritative business systems  inventory—the Defense Information Technology Portfolio Repository  (DITPR)—with the number in its IT budget system, the Select and Native  Programming Data Input System—Information Technology (SNAP-IT),  with the number in the ETP. Further, we reviewed and compared business  system information, such as legacy system migration information in the  ETP, with the information obtained from our recently completed and  ongoing business system reviews to determine whether the information  was consistent. We interviewed BTA officials to discuss any discrepancies.  Furthermore, we obtained and reviewed information from the  Departments of the Air Force, Army, and Navy on the extent to which they  have made progress in satisfying existing recommendations associated  with developing their respective ETPs.
We were unable to determine whether DOD’s fiscal year 2010 information  technology budget submission was prepared in accordance with the  criteria set forth in the Act because the budget submission was not  released in time for us to review for this report. Instead, we analyzed and  compared information contained in the department’s system that is used  to prepare its budget submission (SNAP-IT) with information in the ETP  and DOD’s DITPR system to determine if DOD’s fiscal year 2009 budget  request included all business systems. We interviewed BTA and Assistant  Secretary of Defense (Networks and Information Integration)/Department  of Defense Chief Information Officer (ASD(NII)/DOD CIO) officials to  discuss the accuracy and comprehensiveness of information contained in  the SNAP-IT system, the discrepancies in the information contained in the  ETP, DITPR, and SNAP-IT systems, and efforts under way or planned to  address these discrepancies. DOD officials were not able to provide the  supporting data to address any discrepancies in the number of business  systems contained in DITPR in time for inclusion in our report.
To determine whether DOD has established investment review structures  and processes, we focused on the one Investment Review Board specified  in the Act that we previously reported had yet to be established.  Accordingly, we obtained documentation from and interviewed cognizant  DOD officials about actions completed, under way, and planned relative to  the establishment of the DOD Chief Information Officer Investment  Review Board. We also obtained and reviewed documentation—such as  DOD IT Defense Business Systems Investment Review Process Guidance  and Operation of the Defense Acquisition System, Department of  Defense Instruction Number 5000.02, as well as the Air Force  Information Technology Investment Review Guide and Air Force  Information Technology Portfolio Management and IT Investment  Review—and interviewed knowledgeable DOD officials about efforts to  address DOD corporate and component investment management-related  weaknesses that we identified in previous reports. We also reviewed and  leveraged our previous reports that addressed DOD corporate and  component approaches to managing business system investments.
To determine whether the department was reviewing and approving  business system investments exceeding $1 million, we obtained  information from BTA on the number of defense business systems  certified and approved since our last annual review, including information  about Air Force, Army, and Navy actions that were taken in order to  perform the annual systems reviews as required pursuant to the Act. In  addition, we summarized the results of recent reports associated with  information used during the certification and annual review process. We  also interviewed BTA and ASD(NII)/DOD CIO officials to determine the  steps taken, planned, or under way to validate the accuracy of the  information in DITPR to be used by the review boards in making  certification and approval decisions. In addition, we analyzed selected   business system information contained in DITPR, such as system life cycle  start and end dates, to validate the reliability of the information.
We did not independently validate the reliability of the cost and budget  figures provided by DOD because the specific amounts were not relevant  to our findings. We conducted this performance audit at DOD offices in  Arlington, Virginia, from January 2009 to May 2009, in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objective.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact person named above, key contributors to this  report were Neelaxi Lakhmani (Assistant Director), Justin Booth, Michael  Holland, Anh Le, Emily Longcore, Lee McCracken, Christine San, Sylvia  Shanks, Jennifer Stavros-Turner, and Adam Vodraska.