Information Technology: FBI Is Taking Steps to Develop an Enterprise Architecture, but Much Remains to Be Accomplished
Released: 2005-09-09T00:00:00

Letter

Background

The FBI was founded in 1908 to serve as the primary investigative unit of the Department of Justice. Its missions include protecting the nation from foreign intelligence and terrorist threats, investigating serious federal crimes, providing leadership and assistance to law enforcement agencies, and being responsive to the public in the performance of these duties. Approximately 12,000 special agents and 16,000 mission support personnel are located in the bureau’s Washington, D.C., headquarters and in more than 450 offices in the United States and 45 offices in foreign countries.

Mission responsibilities at the bureau are divided among the following five major organizational components: Counterterrorism and Counterintelligence: identifies, assesses, investigates, and responds to national security threats.

Intelligence: collects, analyzes, and disseminates information on evolving threats to the United States.

Criminal Investigations: investigates serious federal crimes and probes federal statutory violations involving exploitation of the Internet and computer systems.

Law Enforcement Services: provides law enforcement information and forensic services to federal, state, local, and international agencies.

Administration: manages the bureau’s personnel program, budgetary and financial services, records, information resources, and information security.

Each component is headed by an Executive Assistant Director who reports to the Deputy Director, who, in turn, reports to the Director. The components are further organized into 19 subcomponents, such as divisions, offices, and groups. Supporting these subcomponents are various staff offices, including the Office of the CIO. Figure 1 shows a simplified organizational chart of the components, subcomponents, Office of the CIO, and their respective reporting relationships.

The Office of the CIO’s responsibilities include preparing the bureau’s IT strategic plan and operating budget; operating and maintaining existing systems and networks; developing and deploying new systems; defining and implementing IT management policies, procedures, and processes; and developing and maintaining the bureau’s EA. To carry out these responsibilities, the Office of the CIO is organized into four subordinate offices. Figure 2 shows a simplified organizational chart of the CIO’s office, subordinate offices, and their reporting relationships; a brief description of each office’s responsibilities is in table 1. The FBI’s EA program is in the CIO’s Office of IT Policy and Planning.

To execute its mission responsibilities, the FBI has historically relied extensively on IT. For example, it relies on such computerized IT systems as the Combined DNA Index System to support forensic examinations and the National Crime Information Center and the Integrated Automated Fingerprint Identification System to help state and local law enforcement agencies identify criminals. The FBI reports that it collectively manages hundreds of systems, networks, databases, applications, and associated IT tools. As we previously reported, the FBI’s IT environment includes outdated, nonintegrated systems that do not optimally support mission operations.

Following the terrorist attacks of September 11, 2001, the FBI was forced to rethink its mission. As we have reported, this resulted in the bureau shifting its mission focus to detecting and preventing possible future attacks and ultimately led to the FBI’s commitment to reorganize and transform itself. According to the bureau, the complexity of this mission shift, along with the changing law enforcement environment, has strained its existing patchwork of IT systems, which were developed and deployed on an ad hoc basis. The bureau reports that these circumstances will require a major overhaul in its IT systems environment.

To effect this change, the FBI has undertaken an organizational transformation and systems modernization effort. Major goals of the transformation are, among other things, to develop the capability to become a proactive rather than a reactive organization, embrace intelligence as a professional and operational competency, and leverage information across the bureau and with other agencies to “connect the dots.”According to the FBI, an integral part of the transformation will be modernizing the IT systems that support the bureau’s processes. The FBI reports that it will spend approximately $390 million on modernization projects in fiscal year 2005 out of a total IT budget of $737 million. To guide and constrain these and future system modernization investments, the FBI has initiated an effort to align its investments with the new mission being implemented via its transformation. The FBI has stated that a foundational element of this effort is a bureauwide EA.

An EA Is Critical to Successful Systems Modernization

Effective use of EAs, or modernization blueprints, is a trademark of successful public and private organizations. For more than a decade, we have promoted the use of architectures to guide and constrain system modernizations, recognizing them as a crucial means to a challenging goal: agency operational structures that are optimally defined in both business and technological environments. The Congress, the Office of Management and Budget (OMB), and the federal CIO Council have also recognized the importance of an architecture-centric approach to modernization. The Clinger-Cohen Act of 1996 mandates that agency CIOs develop, maintain, and facilitate the implementation of an IT architecture. Further, the E-Government Act of 2002 requires OMB to oversee EA development within and across agencies.

An EA is a systematically derived snapshot—in useful models, diagrams, and narrative—of a given entity’s operations (business and systems), including how its operations are performed, what information and technology are used to perform the operations, where the operations are performed, who performs them, and when and why they are performed. The architecture describes the entity in both logical terms (e.g., interrelated functions, information needs and flows, work locations, systems, and applications) and technical terms (e.g., hardware, software, data, communications, and security). EAs provide these perspectives both for the entity’s current (or “as is”) environment and for its target (or “to be”) environment; they also provide a high-level capital investment roadmap for moving from one environment to the other. In doing so, EAs link organizations’ strategic plans with program implementations.

Among others, OMB, the National Institute of Standards and Technology, and the federal CIO Council have issued frameworks that define the scope and content of architectures. In addition, OMB has since issued a collection of five reference models (Business, Performance, Data/Information, Service, and Technical) that are intended to facilitate governmentwide improvement through cross-agency analysis and the identification of duplicative investments, gaps, and opportunities. While these various frameworks differ in their nomenclatures and modeling approaches, they consistently provide for defining an architecture’s operations in both logical and technical terms and providing these perspectives for both the “as is” and the “to be” environments, as well as the investment roadmap.

Managed properly, an EA can clarify and help to optimize the interdependencies and relationships among an organization’s business operations and the underlying IT infrastructure and applications that support these operations. Employed in concert with other important management controls, such as portfolio-based capital planning and investment control practices, architectures can greatly increase the chances that an organization’s operational and IT environments will be configured to optimize its mission performance. Our experience with federal agencies has shown that making IT investments without defining these investments in the context of an architecture often results in systems that are duplicative, not well integrated, and unnecessarily costly to maintain and interface.

GAO’s EA Management Maturity Framework Is a Tool for Measuring and Improving EA Management Effectiveness

According to guidance published by the federal CIO Council, effective architecture management consists of a number of key practices and conditions. In April 2003, we published a maturity framework that arranges key best practices and conditions of the federal CIO Council’s guide into five hierarchical stages, with Stage 1 representing the least mature and Stage 5 being the most mature. The framework provides an explicit benchmark for gauging the effectiveness of EA management and provides a roadmap for making improvements. Each of the five stages is described below, and the stages and their core elements are shown in table 2. (See app. II for a more detailed description of our framework and associated core elements.) 1. Creating EA awareness. The organization does not have plans to develop and use an architecture, or it has plans that do not demonstrate an awareness of the value of having and using an architecture. While Stage 1 agencies may have initiated some architecture activity, these agencies’ efforts are ad hoc and unstructured, lack institutional leadership and direction, and do not provide the management foundation necessary for successful architecture development. 2. Building the EA management foundation. The organization recognizes that the architecture is a corporate asset by vesting accountability for it in an executive body that represents the entire enterprise. At this stage, an organization assigns architecture management roles and responsibilities and establishes plans for developing architecture products and for measuring program progress and product quality; it also commits the resources necessary for developing an architecture—people, processes, and tools. 3. Developing the EA. The organization focuses on developing architecture products according to the selected framework, methodology, tool, and established management plans. Roles and responsibilities assigned in the previous stage are in place, and resources are being applied to develop actual architecture products. The scope of the architecture has been defined to encompass the entire enterprise, whether organization based or function based. 4. Completing the EA. The organization has completed its architecture products—meaning that the products have been approved by the architecture steering committee or an investment review board and by the CIO. Further, an independent agent has assessed the quality (i.e., completeness and accuracy) of the architecture products. Additionally, evolution of the approved products is governed by a written architecture maintenance policy approved by the head of the organization. 5. Leveraging the EA to manage change. The organization has secured senior leadership approval of the architecture products and has a written institutional policy stating that IT investments must comply with the architecture, unless granted an explicit compliance waiver. Further, decision makers are using the architecture to identify and address ongoing and proposed IT investments that are conflicting, overlapping, not strategically linked, or redundant. Also, the organization tracks and measures architecture benefits or return on investment, and adjustments are continuously made to both the architecture management process and the architecture products.

Our Prior Reviews Have Emphasized the Need for the FBI to Establish Architecture Management Capabilities

Over the past several years, reviews of the FBI’s efforts to leverage IT to support its transformation have identified the bureau’s lack of an EA as a significant management weakness. For example, during 2002, we reported that the FBI did not have an EA. Because our research and experience at federal agencies shows that architectures are an essential ingredient to success for transformations like the FBI’s, we reported that the bureau should establish the management foundation that is necessary to begin successfully developing, implementing, and maintaining an EA.

Between September 2003 and September 2004, we reported on a number of FBI IT transformation challenges, including effectively developing and using an architecture. More specifically, we reported in September 2003 that the bureau had not yet acted on our recommendation for an EA, having only established 1 of the 31 key EA management capabilities described in our architecture management maturity framework, and that this limited capability was due in part to the fact that the architecture’s development was not being treated as an agency priority. Accordingly, we recommended that the Director make architecture development and use a priority, and we provided additional recommendations to help the bureau establish the management capabilities needed to develop, implement, and maintain its architecture. The FBI agreed with our recommendations.

Since we reported on the FBI’s lack of an architecture, others have similarly reported on this gap in the bureau’s ability to effectively modernize its systems and transform its operations. For example, in March 2004, the Department of Justice Inspector General testified that the lack of an architecture was a contributing factor to the continuing cost and schedule shortfalls being experienced by the bureau on its Trilogy investigative case management system, which was the FBI’s centerpiece systems modernization project. Moreover, the National Research Council reported in May 2004 that while the bureau had made significant progress in its IT systems modernization program, the FBI was not on the path to success, in part, because it had not yet developed an EA.

The FBI initiated its current effort to develop an architecture in late 2003. For example, in March 2004, the bureau awarded a $1.2 million firm, fixed- price contract for assistance in developing, maintaining, and implementing an EA. It subsequently awarded the same contractor two fixed-price contracts to provide EA security and integration services. Although these contracts are supporting the Office of the CIO, responsibility for contract management resides with the Office of the Chief Financial Officer.

FBI Has Implemented Some Important EA Management Practices, but It Has Yet to Implement Others

As we previously reported, it is critical that the FBI have and use a well- defined EA to guide and constrain its IT investment decisions. We recommended that in order to effectively develop and implement an architecture, the bureau employ rigorous and disciplined architecture management practices. Such practices form the basis of our architecture management maturity framework. The bureau has thus far implemented most of our framework’s key practices associated with establishing an architecture management foundation, but important foundational practices are still missing. It has also implemented key practices related to developing the architecture; however, most architecture development practices are not yet fully implemented, and virtually all practices that are key to completing and leveraging the architecture for organizational change remain to be implemented. While the bureau’s EA efforts to date represent important progress from where it was in 2003, when we last assessed its efforts, much remains to be accomplished before the FBI will have an effective EA program. Without such a program, the bureau will be challenged in its efforts to effectively and efficiently modernize its systems in a way that minimizes duplication and overlap, maximizes integration, and effectively supports organizational transformation.

In March 2005, the FBI completed an EA baseline report on the status of its “as is” EA activities. The purpose of the report was to, among other things, provide a “high-level snapshot” of where it stood in determining and understanding current bureau business processes and supporting IT structures and systems and how it was managing its ongoing architecture development efforts. In May 2005, the bureau issued a similar report on its “to be” architecture activities. On the basis of these reports, along with other documentation and officials’ statements, we determined that the bureau has satisfied 15 of the 31 core elements specified in our architecture management maturity framework, including 7 Stage 2 elements, all Stage 3 elements, 1 Stage 4 element, and 1 Stage 5 element (see table 3). For the remaining elements, the bureau has efforts planned and under way that are intended to satisfy them.

More specifically, for Stage 2, the bureau has satisfied seven of nine core elements. For example, in early 2004, the bureau established a program office—located in the CIO’s office and headed by a senior level executive— that is responsible for EA development and maintenance, including drafting and executing a program management plan. This program office includes a chief architect and five key senior level architect positions for business, applications, information, technology, and security. The office also has positions that are to perform support functions such as quality assurance, risk management, and configuration control.

The bureau also established an Enterprise Architecture Board that includes senior representation from across all bureau business areas and has assigned the board responsibility for directing, overseeing, and approving the architecture. Minutes of board meetings show that this organization meets about every 2 weeks to oversee EA program progress, provide executive direction, and review and approve EA plans and products. These minutes also show that CIO officials and business area representatives regularly attend the meetings.

In addition, the bureau has developed a number of plans, including a program management plan (dated October 2004). According to these plans, the architecture is to describe the “as is” and “to be” environments, as well as a sequencing plan. Moreover, the plans call for describing the enterprise in terms of business, performance, data, application, and technology. These plans also include a schedule of tasks to be performed, associated milestones, and an estimate of resources (e.g., funding, staffing, contractor assistance) for fiscal years 2004 through 2007. In addition, these plans call for developing performance metrics to measure EA development and execution and provide for establishing management controls, such as risk management, quality assurance, and configuration control, for developing and maintaining the architecture.

Other Stage 2 core elements have yet to be fully addressed. For example, the EA program office does not yet have adequate resources. According to the framework, an organization should have the resources (e.g., funding, human capital) to establish and effectively manage its architecture. According to FBI officials, they have adequate financial resources to fund the program and sufficient contractor assistance, and they have been able to use bureau and contractor personnel to staff most of the 13 program office staff positions. However, core staff positions identified by the bureau have not yet been filled: four of the five key architect positions mentioned earlier are vacant. Bureau officials told us that job announcements have been issued for the four key architect positions, but it has been a challenge finding the right candidates. According to the FBI, failing to have these key staff on board hampers the program office’s ability to perform planned tasks. Having qualified staff serving as the core team is important because without them, the program office does not have the proper knowledge, skills, and abilities to properly execute the EA program, including managing and overseeing its contractors.

In addition, although the FBI has selected a framework to determine the type of architecture products to be developed and has acquired an automated tool to capture the content of its products, the bureau does not have a defined methodology (i.e., the specific steps and methods) documenting how it will develop the products’ content. As stated in our framework, a methodology is important because it defines (and thus permits stakeholders and others to understand) the steps necessary to perform the activities associated with capturing EA content in a coherent, consistent, accountable, and repeatable manner. For this reason, our architecture maturity framework calls for using a methodology in conjunction with an EA framework and automated tool. Collectively, these permit architecture development to occur in an effective and efficient manner.

Instead of a defined methodology, the bureau is relying on a combination of its chief architect’s knowledge and certain documentation, such as an EA alignment plan that describes, among other things, the products to be developed, the order in which they are to be developed, the relationships among products, and analyses that are to be performed to help identify gaps and redundancies in the contents of these products. However, this documentation does not include either the specific steps or methods that explain how the content of products is to be developed and documented. It is important to have a documented methodology that is available to and understood by those engaged in providing EA product content, because without one, there is increased risk that products will be inconsistent, incomplete, and incorrect, and thus require rework.

For Stage 3, the bureau has satisfied all six core elements. In particular, the bureau issued a policy in August 2003 that defines, among other things, the scope of the architecture and identifies the major stakeholders, including their roles and responsibilities.

In addition, the bureau has developed a configuration management plan that defines management structures and processes for identifying, tracking, monitoring, reporting, and auditing changes to the architecture products. The plan establishes a configuration control board and makes the security architect responsible for initiating board meetings and ensuring that audits are conducted as intended. To date, this board has identified and begun tracking such changes. For example, products, including the program management plan, EA principles, “as is” architecture, and EA software tool, have been identified and placed under configuration management in accordance with the plan.

Further, the program office reports that it is in the process of developing its “as is” architecture. According to the March 2005 report, the bureau has issued several iterations of a “high-level” version of its “as is” architecture that describes the bureau’s business, data, application, and technology environments. However, these iterations do not include performance descriptions. Moreover, the other “as is” descriptions are not complete, according to the report. For example, as part of the information/data description, the program office is in the process of completing ongoing efforts to map FBI data to the business processes that use these data. In addition, as part of the application description, the program office is working to develop a system architecture diagram to show how the various IT applications currently interrelate. Also, while the program office has developed a business architecture description, it has not performed a detailed decomposition of the business processes described. The bureau had planned to complete the remaining work on the “as is” architecture by mid-summer.

The office also is in the process of developing the “to be” architecture. According to the FBI’s May 2005 report, the initial version of the “to be” architecture includes business, performance, information/data, service, and technology descriptions. However, the report identifies additional work needed to complete this version. For example, according to the report, the service reference models need to be further defined to provide a detailed framework that supports the transition to the “to be” environment. In addition, the bureau reports that data exchange models need to be developed to provide better understanding of data exchange processes and whether opportunities exist for improvement. Further, the bureau reports that it needs to develop a framework so that it can better understand the relationships among EA components, such as between the business reference model and the service reference model, and between the service reference model and the technology reference model. The bureau plans to issue the next version of its “to be” architecture in fiscal year 2006.

In addition, the bureau reports it has developed a “high level” description of a sequencing plan that is not yet complete; the next version of the plan is scheduled for issuance in September 2005.

Two additional elements (one Stage 4 and one Stage 5 element) have also been satisfied. Specifically, while EA products and processes to date have not been independently verified and validated, the FBI hired a contractor in April 2005 to begin performing such assessments on both the EA products and the processes used to develop them. According to the contract statement of work, the results of these assessments are to be shared with the program office and reported to the steering committee.

Also, the bureau has defined a management structure and process to formally manage EA change. According to its configuration management plan (dated February 3, 2005), the bureau is using an automated tool to manage critical EA work products as they are developed and changed. Further, the bureau established a Change Management Board to resolve critical issues, including those that require a major commitment of resources, vary from the EA strategy, or require a policy change.

Beyond these two elements, 14 core elements in Stages 4 and 5 have yet to be satisfied. In particular, key architecture products have yet to be completed. As previously noted, the bureau is still in the process of developing both its “as is” and “to be” architectures, for example. The sequencing plan is also a work in process. (A summary of the results of our assessment on the FBI’s satisfaction of the core elements for each of the stages are provided in app. III.)

Discussing the bureau’s EA program, the FBI’s CIO said that significant progress has been made, which he attributed to top-level organizational commitment and focus on EA, as well as assignment of bureauwide IT budget control and authority to the CIO. Despite this progress, much remains to be accomplished before the FBI will have an effective EA program. According to our framework, effective architecture management is generally not achieved until an enterprise has a completed and approved architecture that is being effectively maintained and is being used to leverage organizational change and support investment decision making; having these characteristics is equivalent to having satisfied all of the Stage 2 and 3 core elements and many of the Stage 4 and 5 elements.

Until the bureau gets to that stage, it will be challenged in its efforts to implement modernized systems in a way that minimizes overlap and duplication and maximizes integration and mission support. Our prior reviews of federal agencies and research of architecture best practices have shown that attempting to modernize systems without a well-defined and verifiable architecture and associated management capabilities increases the risk that large sums of money and much time and effort will be invested in technology solutions that are duplicative, are not well integrated, are unnecessarily costly to maintain and interface, and do not effectively optimize mission performance.

Bureau Is Not Effectively Managing Its EA Contractor

Federal acquisition regulations and relevant IT acquisition management guidance recognize the importance of effectively managing contractor activities. The Federal Acquisition Regulation (FAR), for example, directs agencies to use performance-based contracting to the maximum extent practicable when acquiring most services. Under the FAR, performance- based contracting includes (1) defining the work to be performed in measurable, results-oriented terms; (2) specifying performance standards (quality and timeliness) that are tied to contractual requirements; (3) having a quality assurance plan that describes how the contractor’s performance in meeting requirements will be measured against standards; and (4) establishing positive and negative contractor performance incentives. The FAR and associated regulations also require government oversight of contracts to ensure that the contractor (the service provider) performs the requirements of the contract, and the government (the service receiver or customer) receives the service as intended. However, the regulations do not prescribe specific methods for this oversight.

Other acquisition management guidance identifies effective contractor tracking and oversight as a key activity and describes a number of practices associated with this activity, including establishing a written policy for contract tracking and oversight, designating responsibility for contract tracking and oversight activities, establishing a group that is responsible for managing contract tracking and oversight activities, and using approved contractor planning documents as a basis for tracking and overseeing the contractor.

The FBI’s approach to managing its EA contract does not include most of the performance-based contracting features described in the FAR. Specifically, although the contract’s statement of work defines when products are due (i.e., timeliness standards), it does not specify the products in results-oriented, measurable terms. For example, the statement of work defines requirements in terms of general product descriptions such as “as is” and “to be” architectures and a sequencing plan. Further, it does not specify quality standards for products and does not define incentives for addressing either timeliness or quality standards.

The bureau also does not have plans for assuring the quality of the contractor’s work. Instead, bureau officials told us that they follow the bureau’s long-standing approach of working with the contractor to determine whether each deliverable is acceptable. As an example, the bureau received a draft of its “as is” architecture on August 22, 2004. According to bureau officials, the draft was of poor quality, and the bureau did not accept it. The bureau then worked with the contractor to improve the quality of the product, and after several iterations, the bureau accepted a draft of the “as is” architecture on September 30, 2004. However, because the bureau did not have either quality standards or a quality assurance plan, the basis for acceptance was not available for us to independently assess.

In tracking and overseeing its contractor, the FBI also has not employed the kind of effective practices specified in relevant acquisition management guidance. For example, the bureau does not have a written policy to govern its tracking and oversight activities, has not designated responsibility or established a group for performing contract tracking and oversight activities, and has not developed an approved contractor monitoring plan. Instead, the bureau holds weekly status meetings with its EA contractor to discuss progress and plans, and it is receiving incremental drafts of work products in an effort to increase visibility into contractor activities and thereby minimize the number of unacceptable deliverables and associated rework.

FBI officials from the offices of the Chief Financial Officer and CIO attributed the current contract management approach to several factors. First, they said that the FBI has historically been challenged in developing statements of work that clearly define requirements and establish performance (quality and timeliness) standards, which are essential to effective performance-based contracting. Second, these officials stated that they are still working to define effective contract management controls. Specifically, as part of the CIO office’s transformation, including implementing its recently assigned agencywide authority and control over IT resources, these officials are developing standard policies and procedures for managing IT. In particular, these policies and procedures are to include an FBI-wide standard life-cycle management directive that is to define procedures for the use of performance-based contracting methods and the establishment of tracking and oversight structures, policies, and processes. The officials told us they began implementing parts of the directive in late June 2005, but added that certain key practices, such as acquisition management, were early drafts and required further development. However, the officials were unable to provide a date for when the drafts would be finalized and implementation of the practices would begin.

In the absence of performance-based contracting and effective tracking and oversight, the bureau’s ability to effectively manage its EA contractor is constrained. This means that the FBI is at risk of taking more time and spending more money than necessary to produce a well-defined architecture.

Conclusions

Having a well-defined and enforced architecture is critical to the FBI’s ability to effectively and efficiently modernize its mission operations and supporting IT environment. The bureau has taken steps aimed at developing such an architecture and has made important progress in doing so; however, much remains to be accomplished before it will have implemented our prior recommendations and established an effective EA program. As it moves forward, it is important for the bureau to employ all the effective architecture management practices that we have previously recommended, and to do so expeditiously. Moreover, given that the FBI’s program is heavily relying on contractor support, it is also important for the bureau to ensure that it employs effective contract management controls that will enable it to, among other things, define contractor work to be performed in measurable, results-oriented terms; establish positive and negative contractor performance incentives; and define and implement contractor tracking and oversight processes consistent with acquisition management guidance. Currently, the FBI does not have such controls in place, and as a result, it is increasing the risk that it will take more time and money to develop a well-defined EA than is necessary. If the bureau does not begin employing the kind of effective contract management controls contained in federal regulations and related guidance, its architecture efforts will continue to be at risk. In turn, its systems modernization will continue to be challenged in its ability to efficiently and effectively support mission operations through modern IT systems.

Recommendations for Executive Action

In light of our prior comprehensive set of recommendations for strengthening the FBI’s EA program, we are not making additional recommendations at this time relative to satisfying the practices embodied in our architecture management maturity framework.

Given the FBI’s heavy reliance on contractor assistance in developing its EA and the state of its contract management controls, we recommend that the FBI Director direct the Chief Financial Officer, in conjunction with the CIO, to ensure that to the maximum extent practicable, performance-based contracting activities, along with effective contract tracking and oversight practices, are employed prospectively on all EA contract actions. This should include, among other things, defining contractor work in measurable, results-oriented terms; establishing positive and negative contractor performance incentives; and defining and implementing contractor tracking and oversight processes consistent with acquisition management guidance.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the CIO and reprinted in appendix IV, the FBI agreed that the bureau had made progress in developing its architecture. The FBI also stated that it appreciated our assessment and feedback on its EA program and that the bureau would continue to strive to develop a robust EA program supported by effective contract management practices. In this regard, the FBI cited steps under way to strengthen its EA management foundation. The FBI also noted our recommendation regarding the use of performance-based contracting, stating that its use of fixed-price contracting for EA support has been successful. We believe the FBI can benefit from increased use of performance-based contracting techniques even under firm, fixed-priced contracts. In this regard, the FBI agreed, stating that our recommendations provide for effective EA contract management practices and that it is now taking steps to increase its use of performance-based contracting. The FBI stated that it is in the process of increasing employee awareness and providing training on the performance-based approach.

We are sending copies of this report to the Chairmen and Ranking Minority Members of the Senate and House Appropriations Committees. We are also sending copies to the Attorney General; the Director, FBI; the Director, OMB; and other interested parties. This report will also be available at no charge on our Web site at http://www.gao.gov.

Should you have any questions about matters discussed in this report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO contacts and staff who made major contributions to this report are listed in appendix V.

Objectives, Scope, and Methodology

As specified in the conference report accompanying the Consolidated Appropriations Act, 2005, our objectives were to determine (1) whether the Federal Bureau of Investigation (FBI) is managing its enterprise architecture (EA) program in accordance with established best practices and (2) what approach the bureau is following to track and oversee its EA contractor, including the use of effective contractual controls.

For the first objective, we reviewed our EA management maturity framework, Version 1.1, which organizes architecture management best practices into five stages of maturity. This framework is based on A Practical Guide to Federal Enterprise Architecture, published by the federal Chief Information Officer (CIO) Council. We compared our framework with the ongoing efforts of the FBI’s EA program. Specifically, we analyzed the bureau’s EA plans and products, including program management and other plans, key architecture principles, work breakdown structures and corresponding milestones, Enterprise Architecture Board charters and meeting minutes, repository strategy, and EA status reports. We also analyzed relevant policies and procedures, including the bureau’s EA Policy and the Information Technology Life Cycle Management Directive. Moreover, we reviewed draft architecture work products, including iterations of the “as is” and “to be” architectures; we did not, however, assess the contents or quality of these architectural work products because they were in varying degrees of completion and subject to ongoing change. Next, we compared our analyses with the EA management maturity framework practices to determine the extent to which the FBI was employing such effective management practices. We also interviewed bureau officials, such as the CIO, the chief architect, and the head of the EA program office.

For the second objective, we first reviewed key federal regulations and best practices and guidance. In particular, we reviewed relevant federal acquisition regulations on effective contract management, including performance-based contracting methods. Additionally, we reviewed the Software Engineering Institute’s Software Acquisition Capability Maturity Model, version 1.02, for key contractor tracking and oversight best practices. We then analyzed EA contract documentation, including task orders, statements of work, and contract modifications. We also interviewed FBI officials, including the contracting office’s technical representative for overseeing the EA contractor, the chief architect, and the head of the EA program office. We interviewed these officials to verify and clarify our understanding of the bureau’s architecture contract management procedures and to determine whether the bureau is employing effective contractual controls. Additionally, we discussed with these officials the cause and impact of the current state of the bureau’s contract management activities and policies.

We performed our work at FBI headquarters in Washington, D.C., from September 2004 to July 2005, in accordance with generally accepted government auditing standards.

Detailed Descriptions of Elements in GAO’s EA Management Maturity Framework

Because the task of developing, maintaining, and implementing an EA is an important, complex, and difficult endeavor, doing so effectively and efficiently requires that rigorous, disciplined management practices be adopted. Such practices form the basis of our EA management maturity framework, which specifies by stages the key architecture management structures, processes, and controls that are embodied in federal guidance and best practices. The five stages and their associated core elements are described below.

At Stage 1, organizations are becoming aware of the value of an EA, but have not yet established the management foundation needed to develop one. Stage 1 has no core elements: by default, an organization that does not satisfy Stage 2 core elements is at Stage 1.

For Stage 2, our framework specifies nine key practices or core elements that are necessary to provide the management foundation for successfully launching and sustaining an architecture effort: Ensure that adequate resources exist. An organization should have the resources (funding, people, tools, and technology) to establish and effectively manage its architecture. This includes identifying and securing adequate funding to support EA activities; hiring and retaining the right people with the proper knowledge, skills, and abilities to plan and execute the EA program; and selecting and acquiring the right tools and technology to support EA activities.

Establish a committee or group representing the enterprise that is responsible for directing, overseeing, or approving the EA. This committee should include executive-level representatives from each line of business, and these representatives should have the authority to commit resources and enforce decisions within their respective organizational units. By establishing this enterprisewide responsibility and accountability, the agency demonstrates its commitment to building the management foundation and obtaining buy-in from across the organization.

Establish a program office that is responsible for EA development and maintenance. This organizational unit should be devoted to the EA program and responsible for developing a management plan and executing the plan. The plan should include a detailed work breakdown structure; resource estimates (e.g., funding, staffing, and training); performance measures; and management controls for developing and maintaining the architecture.

Appoint a chief architect. The chief architect should be responsible and accountable for the EA, supported by the architecture program office, and overseen by the architecture steering committee. The chief architect (in collaboration with the CIO, the architecture steering committee, and the organizational head) is instrumental in obtaining organizational buy-in for the architecture, including support from the business units, as well as in securing resources to support architecture management functions such as risk management, configuration management, quality assurance, and security management.

Use a framework, methodology, and automated tool to develop the architecture. The framework provides a formal structure for representing the EA, while the methodology is the common set of procedures that the enterprise is to follow in developing the architecture products. The automated tool serves as a repository where architectural products are captured, stored, and maintained.

Develop an architecture program management plan. This plan specifies how and when the architecture is to be developed. It includes a detailed work breakdown structure; resource estimates (e.g., funding, staffing, and training); performance measures; and management controls for developing and maintaining the architecture. The plan demonstrates the organization’s commitment to managing architecture development and maintenance as a formal program.

Ensure that EA plans call for describing both the “as is” and “to be” environments in terms of business, performance, information/data, application/service, and technology. An organization’s program management plan should provide for defining and normalizing the current and future architectures in terms relevant to stakeholders from varying organization levels and disciplines.

Ensure that EA plans address security at each layer. Plans should define how the organization will address security as a distinct area of operational and technology emphasis within the context of each layer.

Ensure that EA plans call for developing metrics for measuring EA progress, quality, compliance, and return on investment. Plans should provide for developing metrics and should describe how these will be used to measure (1) progress towards EA goals, (2) the quality of architecture products and management processes, (3) compliance with the architecture, and (4) EA return on investment.

At Stage 3, our framework specifies six core elements that are necessary to focus on architecture development activities: Issue a written and approved organization policy for EA development. A policy defines the scope of the architecture, including the requirement for a description of the current and target architectures, as well as an investment road map or sequencing plan specifying the move between the two.

Ensure that EA products are under configuration management. This involves ensuring that changes to products are identified, tracked, monitored, documented, reported, and audited.

Ensure that EA products describe or will describe both the “as is” and the “to be” environments, as well as a sequencing plan. Consistent with the EA program plans discussed in Stage 2, an organization should ensure that the EA products being developed are enterprisewide in scope and describe both the current and future environments, as well as a sequencing plan for moving from the current to the target environment.

Ensure that EA plans are described or will be described for both environments in terms of business, performance, information/data, application/service, and technology. Products being developed or drafted should begin to address each of the given terms of reference, or include placeholders for later defining the enterprise in these terms.

Ensure that business, performance, information/data, application/service, and technology descriptions address or will address security. This involves ensuring that each EA product (including those describing the “as is” and “to be” environments in terms of business, performance, information/data, application/service, and technology) explicitly describe how enterprise security is being defined and will be implemented.

Ensure that progress against EA plans is measured and reported. To assist in attaining stated EA program goals and objectives, an organization should understand and disclose its progress against plans.

As EA products emerge, their content should be assessed against the plans to ensure that expectations are being met.

At Stage 4, during which organizations focus on architecture completion activities, organizations need to satisfy eight core elements: Issue a written and approved organization policy for EA maintenance. A policy promotes enterprisewide commitment to keeping the EA up to date. It should provide for establishing a process for architecture maintenance, including oversight and control. It should also identify the roles, responsibilities, and relationships of key players in the maintenance process.

Ensure that EA products and management processes undergo independent verification and validation. This core element involves having an independent third party—such as an internal audit function or a contractor that is not involved with any of the architecture development activities—verify and validate that the products were developed in accordance with architecture processes and product standards. Doing so provides organizations with needed assurance of the quality of the architecture.

Ensure that EA products describe both the “as is” and the “to be” environments, as well as a sequencing plan. Consistent with the EA program plans discussed in Stage 2, an organization should ensure that the EA products completely and correctly describe both the “as is” and the “to be” environments of the enterprise and include a sequencing plan for migrating the organization between the two environments.

Ensure that EA products for both environments are described in terms of business, performance, information/data, application/service, and technology. An organization’s EA products should be defined and normalized in terms meaningful to a wide variety of stakeholders, ranging from the organization’s chief executive officer and strategic planners to its technology implementers and operators.

Ensure that business, performance, information/data, application/service, and technology descriptions address security. An organization should explicitly and consistently address security in its business, performance, information/data, application/service, and technology architecture products. Because security permeates every aspect of an organization’s operations, the nature and substance of institutionalized security requirements, controls, and standards should be captured in the EA products.

Ensure that the organization’s chief information officer has approved the current version of the EA. The current version of the organization’s completed EA should be approved by the CIO.

Ensure that a committee or group representing the enterprise or the investment review board has approved the current version of the EA. The current version of the organization’s completed architecture should also be approved either by the EA steering committee or by the investment review board.

Measure and report on the quality of EA products. An organization should ensure that the nature and content of the EA products meet defined quality standards. This core element entails developing a set of metrics and assessing the products against those metrics.

At Stage 5, during which the focus is on architecture maintenance and implementation activities, organizations need to satisfy eight core elements: Issue a written and approved organization policy for information technology (IT) investment compliance with the EA. A policy that governs the implementation of the architecture should be approved by the organization head. The EA policy should augment architecture development and maintenance policies by providing for an institutional EA implementation process that is aligned with the organization’s capital planning and investment control process.

Ensure that the organization has a process to formally manage EA change. A formal process should be defined and implemented for introducing changes to the architecture. This process should recognize both internally and externally prompted change, and it should provide for continuous capture and analysis of change proposals and informed decision making about whether to make changes.

Make the EA an integral component of the IT investment management process. Because the road map defines the IT systems that an organization plans to invest in as it transitions from the “as is” to the “to be” environment, the architecture is a critical frame of reference for making IT investment decisions. Using the architecture when making such decisions is important because organizations should approve only those investments that move the organization toward the “to be” environment, as specified in the road map.

Ensure that EA products are periodically updated. An organization will need to periodically update its EA products depending on the volume and degree of approved changes to the EA.

Ensure that IT investments comply with EA. An organization’s IT investments should be aligned and comply with the applicable components of the current version of the EA, and they should not be selected and approved under the organization’s capital planning and investment control process unless compliance is documented by the investment sponsor and substantiated by the architect assessment team.

Ensure that the organization head has approved the current version of the EA. The current version of the EA should ultimately be approved by the head of the organization.

Measure and report return on EA investment. Like any investment, the architecture should produce a return on investment (i.e., a set of benefits), and this return should be measured and reported in relation to costs. Measuring return on investment is important in order to ensure that expected benefits from the architecture are realized and to share this information with executive decision makers, who can then take corrective action to address deviations from expectations.

Measure and report on compliance with the EA. An organization should define metrics, such as number of compliance waivers requested and number granted, to track compliance. Through such measurement and reporting, relevant trends and anomalies can be identified, and corrective action can be taken.

Assessment of the FBI’s EA Efforts against GAO’s EA Management Maturity Framework

Satisfied?

Agency is aware of EA.

The FBI has acknowledged the need for an EA, and the Director has made its development a management priority.

Adequate resources exist.

According to FBI officials, they have identified the financial and human capital resources needed to effectively manage the bureau’s architecture program. While bureau officials stated they have adequate financial resources to fund the program, including sufficient contractor assistance, four of five core architect positions identified as being needed to staff the program office have not yet been filled.

Committee or group representing the enterprise is responsible for directing, overseeing, or approving EA.

The FBI has established an Enterprise Architecture Board to direct, oversee, and approve the EA. The board includes upper-level management from all the operating units, including the counterterrorism, counterintelligence, and finance divisions. Technical representatives, such as the chief technology officer and chief architect, also serve on this board.

Program office responsible for EA development and maintenance exists.

The FBI has established a program office, called the Enterprise Architecture Unit, which is located in the CIO’s office. The program office is responsible for the development, implementation, and maintenance of the EA.

Chief architect exists.

The FBI has designated a chief architect.

EA is being developed using a framework, methodology, and automated tool.

The FBI initially used the Federal Enterprise Architecture Framework and has since switched to OMB’s five Federal Enterprise Architecture Reference Models. The bureau is using the Popkin System Architect tool. However, the bureau does not have a documented methodology that defines how EA products are to be developed. Instead of a defined methodology, the bureau is relying on a combination of its chief architect’s knowledge and certain documentation, such as an EA alignment plan that describes, among other things, the products to be developed, the order in which they are to be developed, the relationships among products, and analyses that are to be performed to help identify gaps and redundancies in the contents of these products. However, this documentation does not include either the specific steps or methods that explain how the content of products is to be developed and documented.

Satisfied?

EA plans call for describing the “as is” and “to be” environments, and a sequencing plan.

The EA program management plan (dated October 2004) calls for the development of “as is” and “to be” environments as well as a sequencing plan.

EA plans call for describing the enterprise in terms of business, performance, information/data, application/service, and technology.

The FBI’s EA baseline report (dated March 2005) and other plans call for the development of business, performance, data, applications, and technology descriptions.

EA plans call for business, performance, information/data, application/service, and technology descriptions to address security.

The FBI’s EA baseline report (dated March 2005) and other plans call for security services to be defined for each of the descriptions.

EA plans call for developing metrics for measuring EA progress, quality, compliance, and return on investment.

The EA policy (dated August 2003) and program management plan call for developing metrics to measure progress, quality, and return on investment.

Written and approved organization policy exists for EA development.

The FBI has a written policy for EA development (dated August 2003) that was approved and signed by the CIO.

EA products are under configuration management.

The bureau has a configuration management plan that defines management structures and processes for identifying, tracking, monitoring, reporting, and auditing changes to the architecture products. EA products, such as the program management plan, EA principles, initial versions of the “as is” architecture, and EA software tool, have been identified and placed under configuration management in accordance with the plan.

EA products describe or will describe the enterprise’s business, performance, information/data, application/service, and the technology that supports them.

The FBI is in the process of developing its “as is” and “to be” architectures. It reports that to date, it has issued what it describes as “high level” versions of each, but that these versions need additional work to be complete. The initial version of the “to be” includes the enterprise's business, performance, information/data, service, and technology descriptions. The latest draft of the “as is” also includes all of these descriptions, except performance. According to FBI officials, performance was omitted due to an oversight on their part, and they intend to address performance in the next version of the “as is” architecture.

EA products describe or will describe the “as is” and the “to be” environments, and a sequencing plan.

The FBI is in the process of developing its “as is” and “to be” architectures. It reports that to date, it has issued what it describes as “high level” versions of each, but that these versions need additional work to be complete. The FBI also reports that it has developed a “high level” description of a sequencing plan that is not yet complete.

Satisfied?

Business, performance, information/data, application/service, and technology address or will address security.

The FBI is in the process of developing its “as is” and “to be” architectures, as described above. These versions of its architectures include a description of security services. According to FBI officials, these versions are not yet complete.

Progress against EA plans is measured and reported.

The FBI is measuring and reporting progress against EA plans.

Written and approved organization policy exists for EA maintenance.

The FBI does not have a written and approved policy for EA maintenance. While the bureau has an EA development policy, it does not address architecture maintenance, nor does it assign responsibility and accountability for maintenance.

EA products and management processes undergo independent verification and validation.

While EA products and processes to date have not been independently verified and validated, the FBI hired a contractor in April 2005 to begin performing such assessments on both the EA products and the processes used to develop them.

EA products describe the enterprise’s business, performance, information/data, application/service, and the technology that supports them.

Initial EA products describe the enterprise’s business, performance, information/data, application/service, and the technology that supports them. However, the FBI reports that these products are not completed.

EA products describe the “as is” and the “to be” environments, and a transitioning (sequencing) plan.

Initial EA products describe the “as is” and the “to be” environments and a sequencing plan. However, the FBI reports that these products are not completed.

Business, performance, data, application, and technology descriptions address security.

Initial EA products include business, performance, information/data, application/service, and the technology descriptions that address security. However, the FBI reports that these products are not completed.

Organization’s chief information officer has approved current version of EA.

The FBI is in the process of completing its EA, and when completed, the CIO plans to approve it.

Committee or group representing the enterprise or the investment review board has approved current version of EA.

The FBI is in the process of completing its EA, and when completed, the Enterprise Architecture Board plans to approve it.

Quality of EA products is measured and reported.

Although the FBI is in the process of completing its EA products, it is not currently measuring and reporting quality. FBI plans call for the bureau to begin measuring and reporting EA product quality starting in fiscal year 2006.

Written and approved policy exists for IT investment compliance with EA.

The FBI does not have a written and approved policy addressing IT investment compliance with EA.

Satisfied?

Process exists to formally manage EA change.

The FBI configuration management plan defines a process to formally manage EA change. To manage the process, the bureau established a change management board in January 2003. The board reviews and determines whether to approve changes to the current FBI environment.

EA is integral component of IT investment management process.

The FBI is in the process of completing its EA, and thus, it is not yet an integral part of the bureau’s IT investment process.

The FBI is in the process of completing its EA, and when it is complete, bureau plans call for the products to be periodically updated.

IT investments comply with EA.

All IT investments are not evaluated for compliance with a completed EA.

Organization head has approved current version of EA.

The FBI does not yet have a completed EA for the Director to approve.

Return on EA investment is measured and reported.

The FBI is not yet measuring and reporting return on investment.

Compliance with EA is measured and reported.

The FBI is not yet measuring and reporting EA compliance.

To achieve a particular stage includes satisfying the specified elements in the stage plus all elements from previous stages. For example, to achieve Stage 3 requires achieving the Stage 3-specific elements plus those in Stages 1 and 2.

Comments from the Federal Bureau of Investigation

GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, the following people made key contributions to this report: Gary Mountjoy, Assistant Director; Barbara Collier; Lori Martinez; Teresa Neven; and William Wadsworth.