Defense Business Transformation: Sustaining Progress Requires Continuity of Leadership and an Integrated Approach
Released: 2008-02-07T00:00:00

Letter

Background

DOD is one of the largest and most complex organizations in the world. Overhauling its business operations will take many years to accomplish and represents a huge and possibly unprecedented management challenge. Execution of DOD’s operations spans a wide range of defense organizations, including the military departments and their respective major commands and functional activities, numerous large defense agencies and field activities, and various combatant and joint operational commands that are responsible for military operations in specific geographic regions or theaters of operation. To support DOD’s operations, the department performs an assortment of interrelated and interdependent business functions—using thousands of business systems—related to major business areas such as weapon systems management, supply chain management, procurement, health care management, and financial management. The ability of these systems to operate as intended affects the lives of our warfighters both on and off the battlefield.

To address long-standing management problems, we began our high-risk series in 1990 to identify and help resolve serious weaknesses in areas that involve substantial resources and provide critical services to the public. Historically, high-risk areas have been designated because of traditional vulnerabilities related to their greater susceptibility to fraud, waste, abuse, and mismanagement. As our high-risk program has evolved, we have increasingly used the high-risk designation to draw attention to areas associated with broad-based transformation needed to achieve greater economy, efficiency, effectiveness, accountability, and sustainability of selected key government programs and operations. DOD has continued to dominate the high-risk list, bearing responsibility, in whole or in part, for 15 of our 27 high-risk areas. Of the 15 high-risk areas, the 8 DOD-specific high-risk areas cut across all of DOD’s major business areas. Table 1 lists the 8 DOD-specific high-risk areas and the year in which each area was designated as high risk. In addition, DOD shares responsibility for 7 governmentwide high-risk areas.

GAO designated DOD’s approach to business transformation as high risk in 2005 because (1) DOD’s improvement efforts were fragmented, (2) DOD lacked an enterprisewide and integrated business transformation plan, and (3) DOD had not appointed a senior official at the right level with an adequate amount of time and appropriate authority to be responsible for overall business transformation efforts. Collectively, these high-risk areas relate to DOD’s major business operations, which directly support the warfighter, including how servicemembers get paid, the benefits provided to their families, and the availability of and condition of the equipment they use both on and off the battlefield.

DOD’s pervasive business systems and related financial management deficiencies adversely affect its ability to assess resource requirements; control costs; ensure basic accountability; anticipate future costs and claims on the budget; measure performance; maintain funds control; prevent and detect fraud, waste, and abuse; and address pressing management issues. Over the years, DOD initiated numerous efforts to improve its capabilities to efficiently and effectively support management decisionmaking and reporting, with little success. Therefore, we first designated DOD’s business systems modernization and financial management as high-risk areas in 1995, followed by its approach to business transformation in 2005.

Overview of DOD Business Systems Modernization High-Risk Area

The business systems modernization high-risk area is large, complex, and integral to each of the other high-risk areas, as modernized systems are pivotal enablers to addressing longstanding transformation, financial, and other management challenges. DOD reportedly relies on approximately 3,000 business systems to support its business functions. For fiscal year 2007, Congress appropriated approximately $15.7 billion to DOD, and for fiscal year 2008, DOD has requested about $15.9 billion in appropriated funds to operate, maintain, and modernize these business systems and the associated infrastructures, of which approximately $11 billion was requested for the military departments. For years, DOD has attempted to modernize its many systems, and we have provided numerous recommendations to help it do so. For example, in 2001, we provided the department with a set of recommendations to help in developing and using an enterprise architecture (modernization blueprint) and establishing effective investment management controls to guide and constrain how the billions of dollars each year are spent on business systems. We also made numerous project-specific and DOD-wide recommendations aimed at ensuring that the department follows proven best practices when it acquires IT systems and services.

Enterprise Architecture and IT Investment Management Are Two Keys to Successfully Modernizing Systems

Effective use of an enterprise architecture, or modernization blueprint, is a hallmark of successful public and private organizations. For more than a decade, we have promoted the use of architectures to guide and constrain systems modernization, recognizing them as a crucial means to a challenging goal: agency operational structures that are optimally defined in both the business and technological environments. Congress has also recognized the importance of an architecture-centric approach to modernization: the E-Government Act of 2002, for example, requires the Office of Management and Budget (OMB) to oversee the development of enterprise architectures within and across agencies.

In brief, an enterprise architecture provides a clear and comprehensive picture of an entity, whether it is an organization (e.g., a federal department) or a functional or mission area that cuts across more than one organization (e.g., financial management). This picture consists of snapshots of both the enterprise’s current or “As Is” environment and its target or “To Be” environment. These snapshots consist of “views,” which are one or more architecture products (models, diagrams, matrices, text, etc.) that provide logical or technical representations of the enterprise. The architecture also includes a transition or sequencing plan, based on an analysis of the gaps between the “As Is” and “To Be” environments; this plan provides a temporal road map for moving between the two that incorporates such considerations as technology opportunities, marketplace trends, fiscal and budgetary constraints, institutional system development and acquisition capabilities, the dependencies and life expectancies of both new and “legacy” (existing) systems, and the projected value of competing investments. Our experience with federal agencies has shown that investing in IT without defining these investments in the context of an architecture often results in systems that are duplicative, not well integrated, and unnecessarily costly to maintain and interface.

A corporate approach to IT investment management is also characteristic of successful public and private organizations. Recognizing this, Congress developed and enacted the Clinger-Cohen Act in 1996, which requires OMB to establish processes to analyze, track, and evaluate the risks and results of major capital investments in information systems made by executive agencies. In response to the Clinger-Cohen Act and other statutes, OMB developed policy for planning, budgeting, acquisition, and management of federal capital assets and issued guidance. We have also issued guidance in this area, in the form of a framework that lays out a coherent collection of key practices that when implemented in a coordinated manner, can lead an agency through a robust set of analyses and decision points that support effective IT investment management. This framework defines institutional structures, such as investment review boards, and associated processes, such as common investment criteria. Further, our investment management framework recognizes the importance of an enterprise architecture as a critical frame of reference for organizations making IT investment decisions. Specifically, it states that only investments that move the organization toward its target architecture, as defined by its sequencing plan, should be approved (unless a waiver is provided or a decision is made to modify the architecture). Moreover, it states that an organization’s policies and procedures should describe the relationship between its architecture and its investment decision-making authority. Our experience has shown that mature and effective management of IT investments can vastly improve government performance and accountability, and can help to avoid wasteful IT spending and lost opportunities for improvements.

Financial Management

A major component of DOD’s business transformation strategy is its FIAR Plan, issued in December 2005 and updated annually in June and September. The FIAR Plan was issued pursuant to section 376 of the National Defense Authorization Act for Fiscal Year 2006. Section 376 limited DOD’s ability to obligate or expend funds for fiscal year 2006 on financial improvement activities until the department submitted a comprehensive and integrated financial management improvement plan to congressional defense committees. Section 376 required the plan to (1) describe specific actions to be taken to correct deficiencies that impair the department’s ability to prepare timely, reliable, and complete financial management information and (2) systematically tie such actions to process and control improvements and business systems modernization efforts described in the business enterprise architecture and transition plan. The John Warner National Defense Authorization Act for Fiscal Year 2007 continued to limit DOD’s ability to obligate or expend funds for financial improvement until the Secretary of Defense submits a determination to the committees that the activities are consistent with the plan required by section 376.

DOD intends for the FIAR Plan to provide DOD components with a road map for resolving problems affecting the accuracy, reliability, and timeliness of financial information, and obtaining clean financial statement audit opinions. As such, the FIAR Plan greatly depends on the actions taken by DOD components, including efforts to (1) develop and implement systems that are in compliance with DOD’s BEA, (2) implement sustained improvements in business processes and controls to address material weaknesses, and (3) achieve clean financial statement audit opinions. The FIAR Plan uses an incremental approach to structure its process for examining operations, diagnosing problems, planning corrective actions, and preparing for audit. Although the FIAR Plan provides estimated timeframes for achieving auditability in specific areas or components, it does not provide a specific target date for achieving a clean audit opinion on the departmentwide financial statements. Rather, the FIAR Plan recognizes that its ability to fully address DOD’s financial management weaknesses and ultimately achieve clean audit opinions will depend largely on the efforts of its components to successfully implement new business systems on time, within budget, and with the intended capability.

DOD Has Made Progress in Addressing Its Business Transformation Efforts, but Critical Actions Are Needed to Provide Comprehensive, Integrated, and Strategic Planning and Focused, Sustained Leadership

DOD’s leaders have demonstrated a commitment to making the department’s business transformation a priority and made progress in establishing a management framework for these efforts. For example, the Deputy Secretary of Defense has overseen the establishment of various management entities and the creation of plans and tools to help guide business transformation at DOD. However, our analysis has shown that these efforts are largely focused on business systems modernization and that ongoing efforts across the department’s business areas are not adequately integrated. In addition, DOD lacks two crucial features that are integral to successful organizational transformation: (1) a strategic planning process that results in a comprehensive, integrated, and enterprisewide plan or interconnected plans and (2) a senior leader who is responsible and accountable for business transformation and who can provide full-time focus and sustained leadership.

DOD Has Made Progress in Addressing Its Business Transformation Challenges

DOD’s senior leadership has shown commitment to transforming the department’s business operations, and DOD has taken a number of positive steps to begin this effort. Because of the impact of the department’s business operations on its warfighters, DOD recognizes the need to continue working toward transforming its business operations and providing transparency in this process. The department has devoted substantial resources and made important progress toward establishing key management structures and processes to guide business systems investment activities, particularly at the departmentwide level, in response to legislation that codified many of our prior recommendations related to DOD business systems modernization and financial management.

Specifically, in the past few years, DOD has established the Defense Business Systems Management Committee, investment review boards, and the Business Transformation Agency to manage and guide business systems modernization. The Defense Business Systems Management Committee and investment review boards were statutorily required by the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 to review and approve the obligation of funds for defense business systems modernization, depending on the cost and scope of the system in review. The Business Transformation Agency was created to support the top-level management body, the Defense Business Systems Management Committee, and to advance DOD-wide business transformation efforts.

Additionally, DOD has developed a number of tools and plans to enable these management entities to help guide business systems modernization efforts. The tools and plans include the BEA and the ETP. The ETP is currently considered the highest-level plan for DOD business transformation. According to DOD, the ETP is intended to summarize all levels of transition planning information (milestones, metrics, resource needs, and system migrations) as an integrated product for communicating and monitoring progress, resulting in a consistent framework for setting priorities and evaluating plans, programs, and investments.

Our analysis of these tools, plans, and meeting minutes of the various transformational management entities shows that these efforts are largely focused on business systems modernization, and that this framework has yet to be expanded to encompass all of the elements of overall business transformation. Furthermore, DOD has not clearly defined or institutionalized in directives the interrelationships, roles and responsibilities, or accountability for the various entities that make up its management framework for overall business transformation. For example, opinions differ within DOD as to which senior governance body will serve as the primary body responsible for overall business transformation. Some officials stated that the Defense Business Systems Management Committee would serve as the senior-most governance entity, while others stated that the Deputy’s Advisory Working Group, a group that provides departmentwide strategic direction on various issues, should function as the primary decision-making body for business transformation.

Additionally, opinions differ between the two entities regarding the definition of DOD’s key business areas, with the Defense Business Systems Management Committee and the Business Transformation Agency using a broader definition of business processes than that of the Deputy’s Advisory Working Group and its supporting organizations. Until such differences are resolved and the department institutionalizes a management framework that spans all aspects of business transformation, DOD will not be able to integrate related initiatives into a sustainable, enterprisewide approach and to resolve weaknesses in business operations.

Critical Actions Are Needed to Provide Comprehensive, Integrated, and Strategic Planning and Focused, Sustained Leadership for DOD’s Overall Business Transformation Efforts

As we have testified and reported for years, a successful, integrated, departmentwide approach to addressing DOD’s overall business transformation requires two critical elements: a comprehensive, integrated, and enterprisewide plan and an individual capable of providing full-time focus and sustained leadership both within and across administrations, dedicated solely to the integration and execution of the overall business transformation effort.

DOD Lacks a Strategic Planning Process That Results in a Comprehensive, Integrated, and Enterprisewide Plan or Set of Plans

DOD continues to lack a comprehensive, integrated, and enterprisewide plan or set of linked plans for business transformation that is supported by a comprehensive planning process and guides and unifies its business transformation efforts. Our prior work has shown that this type of plan should help set strategic direction for overall business transformation efforts and all key business functions; prioritize initiatives and resources; and monitor progress through the establishment of performance goals, objectives, and rewards. Furthermore, an integrated business transformation plan would be instrumental in establishing investment priorities and guiding the department’s key resource decisions.

While various plans exist for different business areas, DOD’s various business-related plans are not yet integrated to include consistent reporting of goals, measures, and expectations across institutional, unit, and individual program levels. Our analysis shows that plan alignment and integration currently focus on data consistency among plans, meaning that plans are reviewed for errors and inconsistencies in reported information, but there is a lack of consistency in goals and measurements among plans. Other entities such as the Institute for Defense Analyses, the Defense Science Board, and the Defense Business Board have similarly reported the need for DOD to develop an enterprisewide plan to link strategies across the department for transforming all business areas and thus report similar findings.

DOD officials recognize that the department does not have an integrated plan in place, although they have stated that their intention is to expand the scope of the ETP so that it becomes a more robust enterprisewide planning document and to evolve this plan into the centerpiece strategic document. DOD updates the ETP twice a year, once in March as part of DOD’s annual report to Congress and once in September, and DOD has stated the department’s goal is to evolve the plan into a comprehensive, top-level planning document for all business functions. DOD released the most recent ETP update on September 28, 2007, and we will continue to monitor developments in this effort.

The National Defense Authorization Act for Fiscal Year 2008 requires the Secretary of Defense, acting through the CMO, to develop a strategic management plan to include detailed descriptions of such things as performance goals and measures for improving and evaluating the overall efficiency and effectiveness of the business operations of the department, key initiatives to achieve these performance goals, procedures to monitor progress, procedures to review and approve plans and budgets for changes in business operations, and procedures to oversee the development, review, and approval of all budget requests for defense business systems. While these provisions are extremely positive, their impact will depend on DOD’s implementation. We continue to believe that the key to success of any planning process is the extent to which key stakeholders participate, and whether the ultimate plan or set of plans is linked to the department’s overall strategic plan, reflects an integrated approach across the department, identifies performance goals and measures, shows clear linkage to budgets, and ultimately is used to guide business transformation.

Recent Legislation Takes Important Step to Provide Executive-Level Attention to Business Transformation Matters

We have long advocated the importance of establishing CMO positions in government agencies, including DOD, and have previously reported and testified on the key characteristics of the position necessary for success. In our view, transforming DOD’s business operations is necessary for DOD to resolve its weaknesses in the designated high-risk areas and to ensure that the department has sustained leadership to guide its business transformation efforts. Specifically, because of the complexity and long- term nature of business transformation, DOD needs a CMO with significant authority, experience, and a term that would provide sustained leadership and the time to integrate its overall business transformation efforts. Without formally designating responsibility and accountability for results, DOD will face difficulties reconciling competing priorities among various organizations, and prioritizing investments will be difficult and could impede the department’s progress in addressing deficiencies in key business areas.

Clearly, Congress has recognized the need for executive-level attention to business transformation matters and has taken specific action in the National Defense Authorization Act for Fiscal Year 2008 to codify CMO responsibilities at a high level in the department—assigning them to the Deputy Secretary of Defense—as well as other provisions, such as establishing a full-time Deputy CMO and designating CMO responsibilities within the military departments. From a historical perspective, this action is unprecedented and represents significant steps toward giving business transformation high-level management attention. Now that this legislation has been enacted, it will be important for DOD to define the specific roles and responsibilities for the CMO, Deputy CMO, and the service CMOs; ensure clearly delineated reporting relationships among them and other department and service officials; foster good executive-level working relationships for maximum effectiveness; establish appropriate integration and transformation structures and processes; promote individual accountability and performance; and provide for continuity.

Further, in less than 1 year, our government will undergo a change in administrations, which raises questions about continuity of effort and the sustainability of the progress that DOD has made to date. As we have said before, business transformation is a long-term process, and continuity is key to achieving true transformation. One of the challenges now facing DOD, therefore, is establishing this continuity in leadership to sustain progress that has been made to date. In the interest of the department and the American taxpayers, we continue to believe the department needs a full-time CMO over the long term in order to devote the needed focus and continuity of effort to transform its key business operations and avoid billions more in waste each year. As such, we believe the CMO position should be codified as a separate position from the Deputy Secretary of Defense in order to provide full-time attention to business transformation and subject to an extended term appointment. The CMO’s appointment should span administrations to ensure that transformation efforts are sustained across administrations. Because business transformation is a long-term and complex process, a term of at least 5 to 7 years is recommended to provide sustained leadership and accountability.

Moreover, the fact that the National Defense Authorization Act for Fiscal Year 2008 modifies politically appointed positions by codifying a new designation for the Deputy Secretary of Defense, creating a new Deputy Chief Management Officer of DOD, and adding a new designation to the military departments’ under secretary positions to serve as the military departments’ CMOs raises larger questions about succession planning and how the executive branch fills appointed positions, not only within DOD, but throughout the government. Currently, there is no distinction in the political appointment process among the different types of responsibilities inherent in the appointed positions. Further, the positions generally do not require any particular set of management qualifications, even though the appointees may be responsible for non-policy-related functions. For example, appointees could be categorized by the differences in their roles and responsibilities, such as by the following categories: those appointees who have responsibility for various policy issues; those appointees who have leadership responsibility for various operational and management matters; and those appointees who require an appropriate degree of technical competence or professional certification, as well as objectivity and independence (for example, judges, the Comptroller General, and inspectors general).

We have asked for a reexamination of the political appointment process to assess these distinctions as well as which appointee positions should be presidentially appointed and Senate confirmed versus presidentially appointed with advance notification to Congress. For example, those appointees who have policy leadership responsibility could be presidentially appointed and Senate confirmed, while many of those with operational and management responsibility could be presidentially appointed, with a requirement for appropriate congressional notification in advance of appointment. In addition, appropriate qualifications for selected positions, including the possibility of establishing specific statutory qualifications criteria for certain categories of appointees, could be articulated. Finally, the use of term appointments and different compensation schemes for these appointees should be reviewed.

DOD Is Continuing to Improve Its Approach to Modernizing Business Systems, but Challenges Remain

Despite noteworthy progress in establishing institutional business system and management controls, DOD is still not where it needs to be in managing its departmentwide business systems modernization. Until DOD fully defines and consistently implements the full range of business systems modernization management controls (institutional and program- specific), it will not be positioned to effectively and efficiently ensure that its business systems and IT services investments are the right solutions for addressing its business needs, that they are being managed to produce expected capabilities efficiently and cost effectively, and that business stakeholders are satisfied.

For decades, DOD has been attempting to modernize its business systems. We designated DOD’s business systems modernization program as high risk in 1995. Since then, we have made scores of recommendations aimed at strengthening DOD’s institutional approach to modernizing its business systems, and reducing the risks associated with key business system investments. In addition, in recent legislation, Congress included provisions that are consistent with our recommendations, such as in the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005. In response, the department has taken, or is taking, important actions to implement both our recommendations and the legislative requirements and as a result has made noteworthy progress on some fronts in establishing corporate management controls, such as developing a corporate-level BEA, including an ETP, establishing corporate investment management structures and processes, increasing business system life cycle management discipline and leveraging highly skilled staff on its largest business system investments.

However, much more remains to be accomplished to address this high-risk area, particularly with respect to ensuring that effective corporate approaches and controls are extended to and employed within each of DOD’s component organizations (military departments and defense agencies). To this end, our recent work has highlighted challenges that the department still faces in “federating” (i.e., extending) its corporate BEA to its component organizations’ architectures, ensuring that the scope and content of the department’s business systems transition plan addresses DOD’s complete portfolio of IT investments, as well as establishing institutional structures and processes for selecting, controlling, and evaluating business systems investments within each component organization. Beyond this, ensuring that effective system acquisition management controls are actually implemented on each business system investment also remains a formidable challenge, as our recent reports on management weaknesses associated with individual programs have disclosed. Among other things, these reports have identified program- level weaknesses relative to architecture alignment, economic justification, performance management, requirements management, and testing.

DOD Continues to Improve Its Corporate BEA and ETP, but Component Architectures Remain a Challenge

In May 2007, we reported on DOD’s efforts to address a number of provisions in the Fiscal Year 2005 National Defense Authorization Act. Among other things, we stated that the department had adopted an incremental strategy for developing and implementing its architecture, including the transition plan, which was consistent with our prior recommendation and a best practice. We further stated that DOD had addressed a number of the limitations in prior versions of its architecture. However, we also reported that additional steps were needed. Examples of these improvements and remaining issues with the BEA and the ETP are summarized below: The latest version of the BEA contained enterprise-level information about DOD’s “As Is” architectural environment to support business capability gap analyses. As we previously reported, such gap analyses between the “As Is” and the “To Be” environments are essential for the development of a well-defined transition plan.

The latest version included performance metrics for the business capabilities within enterprise priority areas, including actual performance relative to performance targets that are to be met. For example, currently 26 percent of DOD assets are reported by using formats that comply with the Department of the Treasury’s United States Standard General Ledger, as compared to a target of 100 percent. However, the architecture did not describe the actual baseline performance for operational activities, such as for the “Manage Audit and Oversight of Contractor” operational activity. As we have previously reported, performance models are an essential part of any architecture and having defined performance baselines to measure actual performance provides the means for knowing whether the intended mission value to be delivered by each business process is actually being realized.

The latest version identified activities performed at each location/organization and indicates which organizations are or will be involved in each activity. We previously reported that prior versions did not address the locations where specified activities are to occur and that doing so is important because the cost and performance of implemented business operations and technology solutions are affected by the location and therefore need to be examined, assessed, and decided on in an enterprise context rather than in a piecemeal, systems-specific fashion.

The March 2007 ETP continued to identify more systems and initiatives that are to fill business capability gaps and address DOD-wide and component business priorities, and it continues to provide a range of information for each system and initiative in the plan (e.g., budget information, performance metrics, and milestones). However, this version still does not include system investment information for all the defense agencies and combatant commands. Moreover, the plan does not sequence the planned investments based on a range of relevant factors, such as technology opportunities, marketplace trends, institutional system development and acquisition capabilities, legacy and new system dependencies and life expectancies, and the projected value of competing investments. According to DOD officials, they intend to address such limitations in future versions of the transition plan as part of their plans for addressing our prior recommendations. In September 2007, DOD released an updated version of the plan which, according to DOD, continues to provide time-phased milestones, performance metrics, and statement of resource needs for new and existing systems that are part of the BEA and component architectures, and includes a schedule for terminating old systems and replacing them with newer, improved enterprise solutions.

As we have also reported, the latest version of the BEA continues to represent the thin layer of DOD-wide corporate architectural policies, capabilities, rules, and standards. Having this layer is essential to a well- defined federated architecture, but it alone does not provide the total federated family of DOD parent and subsidiary architectures for the business mission area that are needed to comply with the act. The latest version had yet to be augmented by the DOD component organizations’ subsidiary architectures, which are necessary to meeting statutory requirements and the department’s goal of having a federated family of architectures. Under the department’s tiered accountability approach, the corporate BEA focuses on providing tangible outcomes for a limited set of enterprise-level (DOD-wide) priorities, while the components are to define and implement their respective component-level architectures that are aligned with the corporate BEA.

However, we previously reported that well-defined architectures did not yet exist for the military departments, which constitute the largest members of the federation, and the strategy that the department had developed for federating its BEA needed more definition to be executable. In particular, we reported in 2006, that none of the three military departments had fully developed architecture products that describe their respective target architectural environments and developed transition plans for migrating to a target environment, and none was employing the full range of architecture management structures, processes, and controls provided for in relevant guidance. Also, we reported that the federation strategy did not address, among other things, how the component architectures will be aligned with the latest version of the BEA and how it will identify and provide for reuse of common applications and systems across the department.

According to DOD, subsequent releases of the BEA will continue to reflect this federated approach and will define enforceable interfaces to ensure interoperability and information flow to support decision making at the appropriate level. To help ensure this, the BTA plans to have its BEA independent verification and validation contractor examine architecture federation when evaluating subsequent BEA releases. Use of an independent verification and validation agent is an architecture management best practice for identifying architecture strengths and weaknesses. Through the use of such an agent, department and congressional oversight bodies can gain information that they need to better ensure that DOD’s family of architectures and associated transition plan(s) satisfy key quality parameters, such as completeness, consistency, understandability, and usability, which the department’s annual reports have yet to include.

We made recommendations aimed at improving the management and content of the military departments’ respective architectures; ensuring that DOD’s federated BEA provides a more sufficient frame of reference to guide and constrain DOD-wide system investments; and facilitating congressional oversight and promoting departmental accountability through the assessment of the completeness, consistency, understandability, and usability of its federated family business mission area architectures. DOD agreed with these recommendations and has since taken some actions, such as developing an updated version of its federation strategy, which according to DOD officials, addresses some of our recommendations. We have ongoing work for this Subcommittee on the military departments’ architecture programs, and plan to issue a report in early May 2008.

DOD Has Largely Established Key Investment Management Structures, but Related Policies and Procedures at Both the Corporate and Component Levels Are Missing

The department has established and has begun to implement legislatively directed corporate investment review structures and processes needed to effectively manage its business system investments, but it has yet to do so in a manner that is fully consistent with relevant guidance, both at a corporate and component level. To its credit, the department has, for example, established an enterprisewide investment board (Defense Business Systems Management Committee (DBSMC)) and subordinate boards (investment review boards (IRB)) that are responsible for business systems investment governance, documented policies and procedures for ensuring that systems support ongoing and future business needs through alignment with the BEA, and assigned responsibility for ensuring that the information collected about projects meets the needs of DOD’s investment review structures and processes.

However, the department has not developed the full range of project- and portfolio-level policies and procedures needed for effective investment management. For example, policies and procedures do not outline how the DBSMC and IRB investment review processes are to be coordinated with other decision-support processes used at DOD, such as the Joint Capabilities Integration and Development System; the Planning, Programming, Budgeting, and Execution system; and the Defense Acquisition System. Without clear linkages among these processes, inconsistent and uninformed decision making may result. Furthermore, without considering component and corporate budget constraints and opportunities, the IRBs risk making investment decisions that do not effectively consider the relative merits of various projects and systems when funding limitations exist.

Examples of other limitations include not having policies and procedures for (1) specifying how the full range of cost, schedule, and benefit data accessible by the IRBs are to be used in making selection decisions; (2) providing sufficient oversight and visibility into component-level investment management activities, including component reviews of systems in operations and maintenance; (3) defining the criteria to be used for making portfolio selection decisions; (4) creating the portfolio of business system investments; (5) evaluating the performance of portfolio investments; and (6) conducting post implementation reviews of these investments. According to best practices, adequately documenting both the policies and the associated procedures that govern how an organization manages its IT investment portfolio(s) is important because doing so provides the basis for having rigor, discipline, and repeatability in how investments are selected and controlled across the entire organization. Accordingly, we made recommendations aimed at improving the department’s ability to better manage the billions of dollars it invests annually in its business systems and DOD largely agreed with these recommendations but added that while it intends to improve departmental policies and procedures for business system investments, each component is responsible for developing and executing investment management policies and procedures needed to manage the business systems under its tier of responsibility.

According to DOD’s tiered accountability approach, responsibility and accountability for business investment management is tiered, meaning that it is allocated between the DOD corporate level (i.e., Office of the Secretary of Defense) and the components based on the amount of development/modernization funding involved and the investment’s designated tier.

However, as our recent reports show the military departments also have yet to fully develop many of the related policies and procedures needed to execute both project-level and portfolio-level practices called for in relevant guidance for their tier of responsibility. For example, they have developed procedures for identifying and collecting information about their business systems to support investment selection and control, and assigned responsibility for ensuring that the information collected during project identification meets the needs of the investment management process. However, they have yet, for example, to fully document business systems investment policies and procedures for overseeing the management of IT projects and systems and for developing and maintaining complete business systems investment portfolio(s). Specifically, policies and procedures do not specify the processes for decision making during project oversight and do not describe how corrective actions should be taken when the project deviates or varies from the project management plan. Without such policies and procedures, the agency risks investing in systems that are duplicative, stovepiped, nonintegrated, and unnecessarily costly to manage, maintain, and operate. Accordingly, we made recommendations aimed at strengthening the military departments’ business systems management capability, and they largely agreed with these recommendations. Department officials stated that they are aware of the absence of documented policies and procedures in certain areas of project and portfolio-level management, and are currently working on new guidance to address these areas.

Until DOD fully defines departmentwide and component-level policies and procedures for both individual projects and portfolios of projects, it risks selecting and controlling these business systems investments in an inconsistent, incomplete, and ad hoc manner, which in turn reduces the chances that these investments will meet mission needs in the most cost- effective manner.

The department has recently undertaken several initiatives to strengthen business system investment management. For example, it has drafted and intends to shortly begin implementing a new Business Capability Lifecycle approach that is to consolidate management of business system requirements, acquisition, and compliance with architecture disciplines into a single governance process. Further, it has established an Enterprise Integration Directorate in the Business Transformation Agency to support the implementation of enterprise resource planning systems by ensuring that best practices are leveraged and BEA-related business rules and standards are adopted.

Implementing Effective Modernization Management Controls on All Business System Investments Remains a Key Challenge

Beyond establishing the above discussed institutional modernization management controls, such as the BEA, portfolio-based investment management, and system life cycle discipline, the more formidable challenge facing DOD is how well it can implement these and other management controls on each and every business system investment and information technology services outsourcing program. In this regard, we have continued to identify program-specific weaknesses as summarized below.

With respect to taking an architecture-centric and portfolio-based approach to investing in programs, for example, we recently reported that the Army’s approach for investing about $5 billion over the next several years in its General Fund Enterprise Business System, Global Combat Support System-Army Field/Tactical, and Logistics Modernization Program (LMP) did not include alignment with Army enterprise architecture or use of a portfolio-based business system investment review process. Moreover, we reported that the Army did not have reliable processes, such as an independent verification and validation function, or analyses, such as economic analyses, to support its management of these programs. We concluded that until the Army adopts a business system investment management approach that provides for reviewing groups of systems and making enterprise decisions on how these groups will collectively interoperate to provide a desired capability, it runs the risk of investing significant resources in business systems that do not provide the desired functionality and efficiency.

With respect to providing DOD oversight organizations with reliable program performance and progress information, we recently reported that the Navy’s approach for investing in both system and information technology services, such as the Naval Tactical Command Support System (NTCSS) and Navy Marine Corps Intranet (NMCI), had not always met this goal. For NTCSS, we reported that, for example, earned value management, which is a means for determining and disclosing actual performance against budget and schedule estimates, and revising estimates based on performance to date, had not been implemented effectively. We also reported that complete and current reporting of NTCSS progress and problems in meeting cost, schedule, and performance goals had not occurred, leaving oversight entities without the information needed to mitigate risks, address problems, and take corrective action. We concluded that without this information, the Navy cannot determine whether NTCSS, as it was defined and was being developed, was the right solution to meet its strategic business and technological needs. For NMCI, we reported that performance management practices, to include measurement of progress against strategic program goals and reporting to key decision makers on performance against strategic goals and other important program aspects, such as examining service-level agreement satisfaction from multiple vantage points and ensuring customer satisfaction, had not been adequate. We concluded that without a full and accurate picture of program performance, the risk of inadequately informing important NMCI investment management decisions was increased.

Given the program-specific weaknesses that our work has and continues to reveal, it is important for DOD leadership and the Congress to have clear visibility into the performance and progress of the department’s major business system investments. Accordingly, we support the provisions in section 816 of the John Warner National Defense Authorization Act for Fiscal Year 2007 that provide for greater disclosure of business system investment performance to both department and congressional oversight entities, and thus increased accountability for results. More specifically, the legislation establishes certain reporting and oversight requirements for the acquisition of major automated information systems (MAIS) that fail to meet cost, schedule, or performance criteria. In general, a MAIS is a major DOD IT program that is not embedded in a weapon system (e.g., a business system investment). Going forward, the challenge facing the department will be to ensure that these legislative provisions are effectively implemented. To the extent that they are, DOD business systems modernization transparency, oversight, accountability, and results should improve.

We currently have ongoing work for this subcommittee looking at the military departments implementation of a broad range of acquisition management controls, such as architectural alignment, economic justification, and requirements management, on selected business systems at the Departments of the Air Force and Navy.

DOD Has Made Progress in Establishing a Framework for Improving Financial Management Capabilities, but More Work Remains

DOD has taken steps toward developing and implementing a framework for addressing the department’s long-standing financial management weaknesses and improving its capability to provide timely, reliable, and relevant financial information for analysis, decisionmaking, and reporting, a key defense transformation priority. Specifically, this framework, which is discussed in both the department’s ETP and the FIAR Plan is intended to define and put into practice a standard DOD-wide financial management data structure as well as enterprise-level capabilities to facilitate reporting and comparison of financial data across the department. While these efforts should improve the consistency and comparability of DOD’s financial reports, a great deal of work remains before the financial management capabilities of DOD and its components are transformed and the department achieves financial visibility. Examples of work remaining that must be completed as part of DOD component efforts to support the FIAR Plan and ETP include data cleansing; improvements in current policies, processes, procedures, and controls; and implementation of integrated systems. We also note DOD has other financial management initiatives underway, including efforts to move toward performance-based budgeting and to continually improve the reliability of Global War on Terrorism cost reporting.

In 2007, DOD also introduced refinements to its approach for achieving financial statement auditability. While these refinements reflect a clearer understanding of the importance of the sustainability of financial management improvements and the department’s reliance on the successful completion of component (including military services and defense agencies) and subordinate initiatives, they are not without risk. Given the department’s dependency on the efforts of its components to address DOD’s financial management weaknesses, it is imperative that DOD ensure the sufficiency and reliability of (1) corrective actions taken by DOD components to support management attestations as to the reliability of reported financial information; (2) activities taken by DOD components and other initiatives to ensure that corrective actions are directed at supporting improved financial visibility capabilities, beyond providing information primarily for financial statement reporting, and are sustained until a financial statement audit can be performed; and (3) accomplishments and progress reported by DOD components and initiatives.

Key DOD Financial Management Transformation Efforts Recognize the Need for an Integrated Approach

Successful financial transformation of DOD’s financial operations will require a multifaceted, cross-organizational approach that addresses the contribution and alignment of key elements, including strategic plans, people, processes, and technology. DOD uses two key plans, the DOD ETP and the FIAR Plan, to guide transformation of its financial management operations. The ETP focuses on delivering improved capabilities, including financial management, through the deployment of system solutions that comply with DOD and component enterprise architectures. The FIAR Plan focuses on implementing audit-ready financial processes and practices through ongoing and planned efforts to address policy issues, modify financial and business processes, strengthen internal controls, and ensure that new system solutions support the preparation and reporting of auditable financial statements. Both plans recognize that while successful enterprise resource planning system implementations are catalysts for changing organizational structures, improving workflow through business process reengineering, strengthening internal controls, and resolving material weaknesses, improvements can only be achieved through the involvement of business process owners, including financial managers, in defining and articulating their operational needs and requirements and incorporating them, as appropriate, into DOD and component business enterprise architectures. DOD officials have acknowledged that integration between the two initiatives is a continually evolving process. For example, the June 2006 FIAR Plan update stated that some of the department’s initial subordinate plans included only limited integration with Business Transformation Agency initiatives and solutions. According to DOD officials, the use of end-to-end business processes (as provided by its segment approach) to identify and address financial management deficiencies will lead to further integration between the FIAR Plan and ETP.

Two key transformation efforts that reflect an integrated approach toward improving DOD’s financial management capabilities are the Standard Financial Information Structure (SFIS) and the Business Enterprise Information System (BEIS), both of which are discussed in DOD’s ETP and FIAR Plan.

SFIS. Key limitations in the department’s ability to consistently provide timely, reliable, accurate, and relevant information for analysis, decisionmaking, and reporting are (1) its lack of a standard financial management data structure and (2) a reliance on numerous nonautomated data transfers (manual data calls) to accumulate and report financial transactions. In fiscal year 2006, DOD took an important first step toward addressing these weaknesses through publication of its SFIS Phase I data elements and their subsequent incorporation into the DOD BEA. In March 2007, the department issued a checklist for use by DOD components in evaluating their systems for SFIS compliance. SFIS is intended to provide uniformity throughout DOD in reporting on the results of operations, allowing for greater comparability of information. While the first phase of SFIS was focused on financial statement generation, subsequent SFIS phases are intended to provide a standardized financial information structure to facilitate improved cost accounting, analysis, and reporting. According to DOD officials, the department has adopted a two-tiered approach to implement the SFIS data structure. Furthermore, they stated that SFIS is a mandatory data structure that will be embedded into every new financial management system, including enterprise resource planning systems, such as the Army’s General Fund Enterprise Business System and the Air Force’s Defense Enterprise Accounting and Management System (DEAMS). Further, recognizing that many of the current accounting systems will be replaced in the future, the department will utilize a common crosswalk to standardize the data reported by the legacy systems.

BEIS. A second important step that the department took toward improving its capability to provide consistent and reliable financial information for decisionmaking and reporting was to initiate efforts to develop a DOD-level suite of services to provide financial reporting services, cash reporting, and reconciliation services. As an interim solution, financial information obtained from legacy component systems will be cross-walked from a component’s data structure into the SFIS format within BEIS. Newer or target systems, such as DEAMS, will have SFIS imbedded so that the data provided to BEIS will already be in the SFIS format.

According to DOD’s September 2007 FIAR Plan update, the department prepared financial statement reports using SFIS data standards for the Marine Corps general and working capital funds, the Air Force general and working capital funds, and the Navy working capital funds. The department plans to implement SFIS-compliant reporting for the Army working capital funds, the Navy general funds, and its defense agencies in fiscal year 2008. The development and implementation of SFIS and BEIS are positive steps toward standardizing the department’s data structure and expanding its capability to access and utilize data for analysis, management decisionmaking, and reporting, including special reports related to the Global War on Terrorism.

However, it is important to keep in mind that a great deal of work remains. In particular, data cleansing; improvements in policies, processes, procedures, and controls; as well as successful enterprise resource planning system implementations are needed before DOD components and the department fully achieve financial visibility. Our previous reviews of DOD system development efforts have identified instances in which the department faced difficulty in implementing systems on time, within budget, and with the intended capability. For example, as previously noted, the Army continues to struggle in its efforts to ensure that LMP will provide its intended capabilities. In particular, we reported that LMP would not provide the intended capabilities and benefits because of inadequate requirements management and system testing. Further, we found that the Army had not put into place an effective management process to help ensure that the problems with the system were resolved. Until the Army has completed action on our recommendations, it will continue to risk investing billions of dollars in business systems that do not provide the desired functionality or efficiency.

DOD Refines Its Audit Strategy

In fiscal year 2007, DOD introduced key refinements to its strategy for achieving financial statement auditability. These refinements include the following: Requesting audits of entire financial statements rather than attempting to build upon audits of individual financial statement line items.

Focusing on improvements in end-to-end business processes, or segments that underlie the amounts reported on the financial statements.

Using audit readiness validations and annual verification reviews of segment improvements rather than financial statement line item audits to ensure sustainability of corrective actions and improvements.

Forming a working group to begin auditability risk assessments of new financial and mixed systems, such as enterprise resource planning systems, at key decision points in their development and deployment life cycle to ensure that the systems include the processes and internal controls necessary to support repeatable production of auditable financial statements.

To begin implementing its refined strategy for achieving financial statement auditability, DOD modified its business rules for achieving audit readiness to reflect the new approach. Recognizing that a period of time may pass before an entity’s financial statements are ready for audit, the revised business rules provide for an independent validation of improvements with an emphasis on sustaining improvements made through corrective actions. Sustainability of improvements will be verified by DOD components through annual internal control reviews, using OMB’s Circular No. A-123, Appendix A, as guidance.

The department’s move to a segment approach provides greater flexibility in assessing its business processes and in taking corrective actions, if necessary, within defined areas or end-to-end business processes that individually or collectively supports financial accounting and reporting. However, DOD officials recognize that additional guidance is needed in several key areas. For example, DOD has acknowledged that it needs to establish a process to ensure the sufficiency of segment work in providing, individually or collectively, a basis for asserting the reliability of reported financial statement information. DOD officials indicated that they intend to provide additional guidance in this area by March 2008. Additionally, DOD officials acknowledged that a process is needed to ensure that DOD’s annual internal control reviews, including its OMB No. A-123, Appendix A reviews, are properly identifying and reporting on issues, and that appropriate corrective actions are taken when issues are identified during these reviews. To its credit, the department initiated the Check It Campaign in July 2006 to raise awareness throughout the department on the importance on effective internal controls.

Ultimately, DOD’s success in addressing its financial management deficiencies, resolving the long-standing weaknesses that have kept it on GAO’s high-risk list for financial management, and finally achieving financial visibility will depend largely on how well its transformation efforts are integrated throughout the department. Both the ETP and FIAR Plan recognize that successful transformation of DOD’s business operations, including financial management, largely depends on successful implementation of enterprise resource planning systems and processes and other improvements occurring within DOD components. Such dependency, however, is not without risk. To its credit, DOD recently established a working group to begin auditability risk assessments of new financial and mixed systems, such as enterprise resource planning systems. The purpose of these planned assessments is to identify auditability risks that, if not mitigated during the development of the system, may impede the component’s ability to achieve clean audit opinions on its financial statements.

Furthermore, the department has implemented and continually expands its use of a Web-based tool, referred to as the FIAR Planning Tool, to facilitate management, oversight, and reporting of departmental and component efforts. According to DOD officials, the tool is used to monitor progress toward achieving critical milestones identified for each focus area in component initiatives, such as financial improvement plans or accountability improvement plans, or departmentwide initiatives. Given that the FIAR Planning Tool is used to report results to OMB through quarterly update reports to the President’s Management Agenda and to update accomplishments in the FIAR Plan, it is critical that the FIAR Directorate ensure the reliability of reported progress. During a recent meeting with DOD officials, we discussed several areas where FIAR Plan reporting appeared incomplete. Our observations included the following.

FIAR Plan updates, including the 2007 update, do not mention or include the results of audit reports and studies that may have occurred within an update period and how, if at all, any issues identified were addressed. For example, the DOD Inspector General has issued reports in recent years that raise concerns regarding the reliability of the military equipment valuation methodology and the usefulness of the valuation results for purposes beyond financial statement reporting. In 2007, the Air Force Audit Agency also issued reports expressing concerns regarding the reliability of reported military equipment values at Air Force. These audit reports and actions, if any, taken in response to them have not been mentioned to date in updates to the FIAR Plan. Further, although both the June and September 2006 FIAR Plan updates report that an internal verification and validation (IV&V) study was completed to test the military equipment valuation methodology, including completeness and existence of military equipment assets, neither of these reports disclosed the results of the review or corrective actions taken, if any. The absence of relevant audit reports or study results may mislead a reader into believing that no issues have been identified that if not addressed, may adversely affect the results of a particular effort, such as the department’s military equipment valuation initiative. For example, the IV&V study identified several improvements that were needed, in varying degrees, at all the military services and the Special Operations Command in the following areas: (1) documentation of waivers; (2) documentation of support for authorization, receipt, and payment; (3) estimated useful life; and (4) existence of the asset. In its conclusion statement, the IV&V study reported that if the weaknesses identified by the IV&V review are pervasive throughout DOD, the department will have a significant challenge to establish control over its resources and get its military equipment assets properly recorded for a financial statement audit. Recognition of audits and other reviews in the FIAR and subordinate plans would add integrity to reported accomplishments and further demonstrate the department’s commitment to transforming its financial management capabilities and achieving financial visibility.

While the FIAR Plan clearly identifies its dependency on component efforts to achieve financial management improvements and clean financial statement audit opinions, it does not provide a clear understanding of further links or dependency between its subordinate plans, such as between the financial improvement plans, accountability improvement plans, and departmentwide initiatives, such as the military equipment valuation effort. For example, while the 2007 FIAR Plan updates indicate that Army, Navy, and Air Force developed accountability improvement plans that detail steps required for asserting audit readiness on military equipment, they do not clearly articulate the relationship of these plans to other plans, such as component financial improvement plans or the department’s plan to value military equipment. Clear linking of individual plans and initiatives is important to ensuring that efforts occurring at all levels within the department are directed at achieving improved financial visibility in the most efficient and effective manner.

While we are encouraged by DOD’s efforts to implement capabilities that improve comparability of reported financial information, a significant amount of work remains before the department or its components have the capability to provide timely, reliable, and relevant information for all management operations and reporting. We caution the department that going forward it will be important to ensure that its financial management modernization efforts do not become compliance-driven activities resulting in little to no benefit to DOD managers. It is critical that the department ensure that its oversight, management, implementation, and reporting of transformation efforts and accomplishments are focused on the implementation of sustained improvements in DOD’s capability to provide immediate access to accurate and reliable financial information (planning, programming, budgeting, accounting, and cost information) in support of financial accountability and efficient and effective decision making throughout the department.

Mr. Chairman and Members of the Subcommittee, this concludes my statement. I would be happy to answer any questions you may have at this time.

GAO Contact

For questions regarding this testimony, please contact Sharon L. Pickup at (202) 512-9619 or pickups@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement.

Related GAO Products

Organizational Transformation: Implementing Chief Operating Officer/Chief Management Officer Positions in Federal Agencies. GAO-08- 322T. Washington, D.C.: December 13, 2007.

Business Systems Modernization: Air Force Needs to Fully Define Policies and Procedures for Institutionally Managing Investments. GAO- 08-52. Washington, D.C.: October 31, 2007.

Business Systems Modernization: Department of the Navy Needs to Establish Management Structure and Fully Define Policies and Procedures for Institutionally Managing Investments. GAO-08-53. Washington, D.C.: October 31, 2007.

Defense Business Transformation: A Full-time Chief Management Officer with a Term Appointment Is Needed at DOD to Maintain Continuity of Effort and Achieve Sustainable Success. GAO-08-132T. Washington, D.C.: October 16, 2007.

Defense Business Transformation: Achieving Success Requires a Chief Management Officer to Provide Focus and Sustained Leadership. GAO- 07-1072. Washington, D.C.: September 5, 2007.

DOD Business Transformation: Lack of an Integrated Strategy Puts the Army’s Asset Visibility System Investments at Risk. GAO-07-860 Washington, D.C.: July 27, 2007.

DOD Business Systems Modernization: Progress Continues to Be Made in Establishing Corporate Management Controls, but Further Steps Are Needed. GAO-07-733. Washington, D.C.: May 14, 2007.

Business Systems Modernization: DOD Needs to Fully Define Policies and Procedures for Institutionally Managing Investments. GAO-07-538. Washington, D.C.: May 11, 2007.

High-Risk Series: An Update. GAO-07-310. Washington, D.C.: January 2007.

Information Technology: DOD Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting Goals and Satisfying Customers. GAO-07- 51. Washington, D.C.: December 8, 2006.

Defense Business Transformation: A Comprehensive Plan, Integrated Efforts, and Sustained Leadership Are Needed to Assure Success. GAO- 07-229T. Washington, D.C.: November 16, 2006.

Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging Architectures for Organizational Transformation. GAO-06- 831. Washington, D.C.: August 14, 2006.

Department of Defense: Sustained Leadership Is Critical to Effective Financial and Business Management Transformation. GAO-06-1006T. Washington, D.C.: August 3, 2006.

Business Systems Modernization: DOD Continues to Improve Institutional Approach, but Further Steps Needed. GAO-06-658. Washington, D.C.: May 15, 2006.

DOD Systems Modernization: Planned Investment in the Navy Tactical Command Support System Needs to be Reassessed. GAO-06-215. Washington, D.C.: December 5, 2005.

DOD Business Systems Modernization: Important Progress Made in Establishing Foundational Architecture Products and Investment Management Practices, but Much Work Remains. GAO-06-219. Washington, D.C.: November 23, 2005.

Defense Management: Additional Actions Needed to Enhance DOD’s Risk-Based Approach for Making Resource Decisions. GAO-06-13. Washington, D.C.: November 15, 2005.

Defense Management: Foundational Steps Being Taken to Manage DOD Business Systems Modernization, but Much Remains to be Accomplished to Effect True Business Transformation. GAO-06-234T. Washington, D.C.: November 9, 2005. 21st Century Challenges: Transforming Government to Meet Current and Emerging Challenges. GAO-05-830T. Washington, D.C.: July 13, 2005.

DOD Business Transformation: Sustained Leadership Needed to Address Long-standing Financial and Business Management Problems. GAO-05- 723T. Washington, D.C.: June 8, 2005.

Defense Management: Key Elements Needed to Successfully Transform DOD Business Operations. GAO-05-629T. Washington, D.C.: April 28, 2005.

Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity. GAO-04-394G. Washington, D.C.: March 2004.

Information Technology: A Framework for Assessing and Improving Enterprise Architecture Management (Version 1.1). GAO-03-584G. Washington, D.C.: April 2003.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.