Information Technology: FBI Is Taking Steps to Develop an Enterprise Architecture, but Much Remains to Be Accomplished
Published: 2005-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Federal Bureau of Investigation (FBI) is currently modernizing its information technology (IT) systems to support its efforts to adopt a more bureauwide, integrated approach to performing its mission. A key element of such systems modernization programs is the use of an enterprise architecture (EA), which is a blueprint of an agency's current and planned operating and systems environment, as well as an IT investment plan for transitioning between the two. The conference report accompanying FBI's fiscal year 2005 appropriations directed GAO to determine (1) whether the FBI is managing its EA program in accordance with established best practices and (2) what approach the bureau is following to track and oversee its EA contractor, including the use of effective contractual controls.
section_title: What GAO Found
paragraphs: The FBI is managing its EA program in accordance with many best practices, but other such practices have yet to be adopted. These best practices, which are described in GAO's EA management maturity framework, are those necessary for an organization to have an effective architecture program. Examples of practices that the bureau has implemented include establishing a program office that is responsible for developing the architecture, having a written and approved policy governing architecture development, and continuing efforts to develop descriptions of the FBI's "as is" and "to be" environments and sequencing plan. The establishment of these and other practices represents important progress from the bureau's status 2 years ago, when GAO reported that the FBI lacked both an EA and the means to develop and enforce one. Notwithstanding this progress, much remains to be accomplished before the FBI will have an effective EA program. For example, the EA program office does not yet have adequate resources, and the architecture products needed to adequately describe either the current or the future architectural environments have not been completed. Until the bureau has a complete and enforceable EA, it remains at risk of developing systems that do not effectively and efficiently support mission operations and performance. The FBI is relying heavily on contractor support to develop its EA; however, it has not employed effective contract management controls in doing so. Specifically, the bureau has not used performance-based contracting, an approach that is required by federal acquisition regulations whenever practicable. Further, the bureau is not employing the kind of effective contractor tracking and oversight practices specified in relevant acquisition management guidance. According to FBI officials, the agency's approach to managing its EA contractor is based on its long-standing approach to managing IT contractors: that is, working with the contractor on iterations of each deliverable until the bureau deems it acceptable. This approach, in GAO's view, is not effective and efficient. According to FBI officials, as soon as the bureau completes an ongoing effort to redefine its policies and procedures for managing IT programs (including, for example, the use of performance-based contracting methods and the tracking and oversight of contractor performance), it will adopt these new policies and procedures. Until effective contractor management policies and procedures are defined and implemented on the EA program, the likelihood of the FBI effectively and efficiently producing a complete and enforceable architecture is diminished.

section_title: Letter
paragraphs: The Federal Bureau of Investigation (FBI) is attempting to replace much of  its 1980’s-based information technology (IT) systems environment to better  support its plans for an integrated bureauwide approach to performing  critical mission operations, including terrorism prevention and federal  crime investigation. Our research and experience in reviewing federal  agency system modernization programs, including the FBI’s, shows that  attempting such programs without a well-defined and enforceable  enterprise architecture (EA) results in nonintegrated, stand-alone systems  that are duplicative and do not effectively and efficiently support mission  performance.
In September 2003, we reported that the FBI needed an EA to guide its  modernization activities and recommended that the FBI Director designate  the development of a complete architecture as a bureauwide priority and  manage the effort accordingly. In response, the FBI initiated efforts to  accomplish this goal, including hiring a contractor to assist the bureau in  this endeavor. Because of the importance of the EA to the FBI’s  modernization program, the conference report accompanying the fiscal  year 2005 Consolidated Appropriations Act directed us to determine  (1) whether the FBI is managing its EA program in accordance with  established best practices and (2) what approach the bureau is following to  track and oversee its EA contractor, including the use of effective  contractual controls. Details of our objectives, scope, and methodology are  in appendix I.
subsections: section_title: Background
paragraphs: The FBI was founded in 1908 to serve as the primary investigative unit of  the Department of Justice. Its missions include protecting the nation from  foreign intelligence and terrorist threats, investigating serious federal  crimes, providing leadership and assistance to law enforcement agencies,  and being responsive to the public in the performance of these duties.  Approximately 12,000 special agents and 16,000 mission support personnel  are located in the bureau’s Washington, D.C., headquarters and in more  than 450 offices in the United States and 45 offices in foreign countries.
Mission responsibilities at the bureau are divided among the following five  major organizational components:  Counterterrorism and Counterintelligence: identifies, assesses,  investigates, and responds to national security threats.
Intelligence: collects, analyzes, and disseminates information on  evolving threats to the United States.
Criminal Investigations: investigates serious federal crimes and  probes federal statutory violations involving exploitation of the Internet  and computer systems.
Law Enforcement Services: provides law enforcement information and  forensic services to federal, state, local, and international agencies.
Administration: manages the bureau’s personnel program, budgetary  and financial services, records, information resources, and information  security.
Each component is headed by an Executive Assistant Director who reports  to the Deputy Director, who, in turn, reports to the Director. The  components are further organized into 19 subcomponents, such as  divisions, offices, and groups. Supporting these subcomponents are various  staff offices, including the Office of the CIO. Figure 1 shows a simplified  organizational chart of the components, subcomponents, Office of the CIO,  and their respective reporting relationships.
The Office of the CIO’s responsibilities include preparing the bureau’s IT  strategic plan and operating budget; operating and maintaining existing  systems and networks; developing and deploying new systems; defining  and implementing IT management policies, procedures, and processes; and  developing and maintaining the bureau’s EA. To carry out these  responsibilities, the Office of the CIO is organized into four subordinate  offices. Figure 2 shows a simplified organizational chart of the CIO’s office,  subordinate offices, and their reporting relationships; a brief description of  each office’s responsibilities is in table 1. The FBI’s EA program is in the  CIO’s Office of IT Policy and Planning.
To execute its mission responsibilities, the FBI has historically relied  extensively on IT. For example, it relies on such computerized IT systems  as the Combined DNA Index System to support forensic examinations and  the National Crime Information Center and the Integrated Automated  Fingerprint Identification System to help state and local law enforcement  agencies identify criminals. The FBI reports that it collectively manages  hundreds of systems, networks, databases, applications, and associated IT  tools. As we previously reported, the FBI’s IT environment includes  outdated, nonintegrated systems that do not optimally support mission  operations.
Following the terrorist attacks of September 11, 2001, the FBI was forced  to rethink its mission. As we have reported, this resulted in the bureau  shifting its mission focus to detecting and preventing possible future  attacks and ultimately led to the FBI’s commitment to reorganize and  transform itself. According to the bureau, the complexity of this mission  shift, along with the changing law enforcement environment, has strained  its existing patchwork of IT systems, which were developed and deployed  on an ad hoc basis. The bureau reports that these circumstances will  require a major overhaul in its IT systems environment.
To effect this change, the FBI has undertaken an organizational  transformation and systems modernization effort. Major goals of the  transformation are, among other things, to develop the capability to  become a proactive rather than a reactive organization, embrace  intelligence as a professional and operational competency, and leverage  information across the bureau and with other agencies to “connect the  dots.”According to the FBI, an integral part of the transformation will be  modernizing the IT systems that support the bureau’s processes. The FBI  reports that it will spend approximately $390 million on modernization  projects in fiscal year 2005 out of a total IT budget of $737 million. To guide  and constrain these and future system modernization investments, the FBI  has initiated an effort to align its investments with the new mission being  implemented via its transformation. The FBI has stated that a foundational  element of this effort is a bureauwide EA.
subsections: section_title: An EA Is Critical to Successful Systems Modernization
paragraphs: Effective use of EAs, or modernization blueprints, is a trademark of  successful public and private organizations. For more than a decade, we  have promoted the use of architectures to guide and constrain system  modernizations, recognizing them as a crucial means to a challenging goal:  agency operational structures that are optimally defined in both business  and technological environments. The Congress, the Office of Management  and Budget (OMB), and the federal CIO Council have also recognized the  importance of an architecture-centric approach to modernization. The  Clinger-Cohen Act of 1996 mandates that agency CIOs develop, maintain,  and facilitate the implementation of an IT architecture. Further, the  E-Government Act of 2002 requires OMB to oversee EA development  within and across agencies.
An EA is a systematically derived snapshot—in useful models, diagrams,  and narrative—of a given entity’s operations (business and systems),  including how its operations are performed, what information and  technology are used to perform the operations, where the operations are  performed, who performs them, and when and why they are performed.  The architecture describes the entity in both logical terms (e.g.,  interrelated functions, information needs and flows, work locations,  systems, and applications) and technical terms (e.g., hardware, software,  data, communications, and security). EAs provide these perspectives both  for the entity’s current (or “as is”) environment and for its target (or “to  be”) environment; they also provide a high-level capital investment  roadmap for moving from one environment to the other. In doing so, EAs  link organizations’ strategic plans with program implementations.
Among others, OMB, the National Institute of Standards and Technology,  and the federal CIO Council have issued frameworks that define the scope  and content of architectures. In addition, OMB has since issued a  collection of five reference models (Business, Performance,  Data/Information, Service, and Technical) that are intended to facilitate  governmentwide improvement through cross-agency analysis and the  identification of duplicative investments, gaps, and opportunities. While  these various frameworks differ in their nomenclatures and modeling  approaches, they consistently provide for defining an architecture’s  operations in both logical and technical terms and providing these  perspectives for both the “as is” and the “to be” environments, as well as  the investment roadmap.
Managed properly, an EA can clarify and help to optimize the  interdependencies and relationships among an organization’s business  operations and the underlying IT infrastructure and applications that  support these operations. Employed in concert with other important  management controls, such as portfolio-based capital planning and  investment control practices, architectures can greatly increase the  chances that an organization’s operational and IT environments will be  configured to optimize its mission performance. Our experience with  federal agencies has shown that making IT investments without defining  these investments in the context of an architecture often results in systems  that are duplicative, not well integrated, and unnecessarily costly to  maintain and interface.
section_title: GAO’s EA Management Maturity Framework Is a Tool for Measuring and Improving EA Management Effectiveness
paragraphs: According to guidance published by the federal CIO Council, effective  architecture management consists of a number of key practices and  conditions. In April 2003, we published a maturity framework that  arranges key best practices and conditions of the federal CIO Council’s  guide into five hierarchical stages, with Stage 1 representing the least  mature and Stage 5 being the most mature. The framework provides an  explicit benchmark for gauging the effectiveness of EA management and  provides a roadmap for making improvements. Each of the five stages is  described below, and the stages and their core elements are shown in table  2. (See app. II for a more detailed description of our framework and  associated core elements.) 1. Creating EA awareness. The organization does not have plans to  develop and use an architecture, or it has plans that do not demonstrate  an awareness of the value of having and using an architecture. While  Stage 1 agencies may have initiated some architecture activity, these  agencies’ efforts are ad hoc and unstructured, lack institutional  leadership and direction, and do not provide the management  foundation necessary for successful architecture development. 2. Building the EA management foundation. The organization  recognizes that the architecture is a corporate asset by vesting  accountability for it in an executive body that represents the entire  enterprise. At this stage, an organization assigns architecture  management roles and responsibilities and establishes plans for  developing architecture products and for measuring program progress  and product quality; it also commits the resources necessary for  developing an architecture—people, processes, and tools. 3. Developing the EA. The organization focuses on developing  architecture products according to the selected framework,  methodology, tool, and established management plans. Roles and  responsibilities assigned in the previous stage are in place, and  resources are being applied to develop actual architecture products.  The scope of the architecture has been defined to encompass the entire  enterprise, whether organization based or function based. 4. Completing the EA. The organization has completed its architecture  products—meaning that the products have been approved by the  architecture steering committee or an investment review board and by  the CIO. Further, an independent agent has assessed the quality (i.e.,  completeness and accuracy) of the architecture products. Additionally,  evolution of the approved products is governed by a written  architecture maintenance policy approved by the head of the  organization. 5. Leveraging the EA to manage change. The organization has secured  senior leadership approval of the architecture products and has a  written institutional policy stating that IT investments must comply  with the architecture, unless granted an explicit compliance waiver.  Further, decision makers are using the architecture to identify and  address ongoing and proposed IT investments that are conflicting,  overlapping, not strategically linked, or redundant. Also, the  organization tracks and measures architecture benefits or return on  investment, and adjustments are continuously made to both the  architecture management process and the architecture products.
section_title: Our Prior Reviews Have Emphasized the Need for the FBI to Establish Architecture Management Capabilities
paragraphs: Over the past several years, reviews of the FBI’s efforts to leverage IT to  support its transformation have identified the bureau’s lack of an EA as a  significant management weakness. For example, during 2002, we reported  that the FBI did not have an EA. Because our research and experience at  federal agencies shows that architectures are an essential ingredient to  success for transformations like the FBI’s, we reported that the bureau  should establish the management foundation that is necessary to begin  successfully developing, implementing, and maintaining an EA.
Between September 2003 and September 2004, we reported on a number  of FBI IT transformation challenges, including effectively developing and  using an architecture. More specifically, we reported in September 2003  that the bureau had not yet acted on our recommendation for an EA, having  only established 1 of the 31 key EA management capabilities described in  our architecture management maturity framework, and that this limited  capability was due in part to the fact that the architecture’s development  was not being treated as an agency priority. Accordingly, we recommended  that the Director make architecture development and use a priority, and we  provided additional recommendations to help the bureau establish the  management capabilities needed to develop, implement, and maintain its  architecture. The FBI agreed with our recommendations.
Since we reported on the FBI’s lack of an architecture, others have  similarly reported on this gap in the bureau’s ability to effectively  modernize its systems and transform its operations. For example, in March  2004, the Department of Justice Inspector General testified that the lack  of an architecture was a contributing factor to the continuing cost and  schedule shortfalls being experienced by the bureau on its Trilogy  investigative case management system, which was the FBI’s centerpiece  systems modernization project. Moreover, the National Research Council  reported in May 2004 that while the bureau had made significant progress  in its IT systems modernization program, the FBI was not on the path to  success, in part, because it had not yet developed an EA.
The FBI initiated its current effort to develop an architecture in late 2003.  For example, in March 2004, the bureau awarded a $1.2 million firm, fixed- price contract for assistance in developing, maintaining, and implementing  an EA. It subsequently awarded the same contractor two fixed-price  contracts to provide EA security and integration services. Although these  contracts are supporting the Office of the CIO, responsibility for contract  management resides with the Office of the Chief Financial Officer.
section_title: FBI Has Implemented Some Important EA Management Practices, but It Has Yet to Implement Others
paragraphs: As we previously reported, it is critical that the FBI have and use a well- defined EA to guide and constrain its IT investment decisions. We  recommended that in order to effectively develop and implement an  architecture, the bureau employ rigorous and disciplined architecture  management practices. Such practices form the basis of our architecture  management maturity framework. The bureau has thus far implemented  most of our framework’s key practices associated with establishing an  architecture management foundation, but important foundational practices  are still missing. It has also implemented key practices related to  developing the architecture; however, most architecture development  practices are not yet fully implemented, and virtually all practices that are  key to completing and leveraging the architecture for organizational  change remain to be implemented. While the bureau’s EA efforts to date  represent important progress from where it was in 2003, when we last  assessed its efforts, much remains to be accomplished before the FBI will  have an effective EA program. Without such a program, the bureau will be  challenged in its efforts to effectively and efficiently modernize its systems  in a way that minimizes duplication and overlap, maximizes integration,  and effectively supports organizational transformation.
In March 2005, the FBI completed an EA baseline report on the status of its  “as is” EA activities. The purpose of the report was to, among other  things, provide a “high-level snapshot” of where it stood in determining and  understanding current bureau business processes and supporting IT  structures and systems and how it was managing its ongoing architecture  development efforts. In May 2005, the bureau issued a similar report on its  “to be” architecture activities. On the basis of these reports, along with  other documentation and officials’ statements, we determined that the  bureau has satisfied 15 of the 31 core elements specified in our architecture  management maturity framework, including 7 Stage 2 elements, all Stage 3  elements, 1 Stage 4 element, and 1 Stage 5 element (see table 3). For the  remaining elements, the bureau has efforts planned and under way that are  intended to satisfy them.
More specifically, for Stage 2, the bureau has satisfied seven of nine core  elements. For example, in early 2004, the bureau established a program  office—located in the CIO’s office and headed by a senior level executive— that is responsible for EA development and maintenance, including  drafting and executing a program management plan. This program office  includes a chief architect and five key senior level architect positions for  business, applications, information, technology, and security. The office  also has positions that are to perform support functions such as quality  assurance, risk management, and configuration control.
The bureau also established an Enterprise Architecture Board that includes  senior representation from across all bureau business areas and has  assigned the board responsibility for directing, overseeing, and approving  the architecture. Minutes of board meetings show that this organization  meets about every 2 weeks to oversee EA program progress, provide  executive direction, and review and approve EA plans and products. These  minutes also show that CIO officials and business area representatives  regularly attend the meetings.
In addition, the bureau has developed a number of plans, including a  program management plan (dated October 2004). According to these plans,  the architecture is to describe the “as is” and “to be” environments, as well  as a sequencing plan. Moreover, the plans call for describing the enterprise  in terms of business, performance, data, application, and technology. These  plans also include a schedule of tasks to be performed, associated  milestones, and an estimate of resources (e.g., funding, staffing, contractor  assistance) for fiscal years 2004 through 2007. In addition, these plans call  for developing performance metrics to measure EA development and  execution and provide for establishing management controls, such as risk  management, quality assurance, and configuration control, for developing  and maintaining the architecture.
Other Stage 2 core elements have yet to be fully addressed. For example,  the EA program office does not yet have adequate resources. According to  the framework, an organization should have the resources (e.g., funding,  human capital) to establish and effectively manage its architecture.  According to FBI officials, they have adequate financial resources to fund  the program and sufficient contractor assistance, and they have been able  to use bureau and contractor personnel to staff most of the 13 program  office staff positions. However, core staff positions identified by the bureau  have not yet been filled: four of the five key architect positions mentioned  earlier are vacant. Bureau officials told us that job announcements have  been issued for the four key architect positions, but it has been a challenge  finding the right candidates. According to the FBI, failing to have these key  staff on board hampers the program office’s ability to perform planned  tasks. Having qualified staff serving as the core team is important because  without them, the program office does not have the proper knowledge,  skills, and abilities to properly execute the EA program, including  managing and overseeing its contractors.
In addition, although the FBI has selected a framework to determine the  type of architecture products to be developed and has acquired an  automated tool to capture the content of its products, the bureau does not  have a defined methodology (i.e., the specific steps and methods)  documenting how it will develop the products’ content. As stated in our  framework, a methodology is important because it defines (and thus  permits stakeholders and others to understand) the steps necessary to  perform the activities associated with capturing EA content in a coherent,  consistent, accountable, and repeatable manner. For this reason, our  architecture maturity framework calls for using a methodology in  conjunction with an EA framework and automated tool. Collectively, these  permit architecture development to occur in an effective and efficient  manner.
Instead of a defined methodology, the bureau is relying on a combination of  its chief architect’s knowledge and certain documentation, such as an EA  alignment plan that describes, among other things, the products to be  developed, the order in which they are to be developed, the relationships  among products, and analyses that are to be performed to help identify  gaps and redundancies in the contents of these products. However, this  documentation does not include either the specific steps or methods that  explain how the content of products is to be developed and documented. It  is important to have a documented methodology that is available to and  understood by those engaged in providing EA product content, because  without one, there is increased risk that products will be inconsistent,  incomplete, and incorrect, and thus require rework.
For Stage 3, the bureau has satisfied all six core elements. In particular, the  bureau issued a policy in August 2003 that defines, among other things, the  scope of the architecture and identifies the major stakeholders, including  their roles and responsibilities.
In addition, the bureau has developed a configuration management plan  that defines management structures and processes for identifying, tracking,  monitoring, reporting, and auditing changes to the architecture products.  The plan establishes a configuration control board and makes the security  architect responsible for initiating board meetings and ensuring that audits  are conducted as intended. To date, this board has identified and begun  tracking such changes. For example, products, including the program  management plan, EA principles, “as is” architecture, and EA software tool,  have been identified and placed under configuration management in  accordance with the plan.
Further, the program office reports that it is in the process of developing its  “as is” architecture. According to the March 2005 report, the bureau has  issued several iterations of a “high-level” version of its “as is” architecture  that describes the bureau’s business, data, application, and technology  environments. However, these iterations do not include performance  descriptions. Moreover, the other “as is” descriptions are not complete,  according to the report. For example, as part of the information/data  description, the program office is in the process of completing ongoing  efforts to map FBI data to the business processes that use these data. In  addition, as part of the application description, the program office is  working to develop a system architecture diagram to show how the various  IT applications currently interrelate. Also, while the program office has  developed a business architecture description, it has not performed a  detailed decomposition of the business processes described. The bureau  had planned to complete the remaining work on the “as is” architecture by  mid-summer.
The office also is in the process of developing the “to be” architecture.  According to the FBI’s May 2005 report, the initial version of the “to be”  architecture includes business, performance, information/data, service,  and technology descriptions. However, the report identifies additional  work needed to complete this version. For example, according to the  report, the service reference models need to be further defined to provide a  detailed framework that supports the transition to the “to be” environment.  In addition, the bureau reports that data exchange models need to be  developed to provide better understanding of data exchange processes and  whether opportunities exist for improvement. Further, the bureau reports  that it needs to develop a framework so that it can better understand the  relationships among EA components, such as between the business  reference model and the service reference model, and between the service  reference model and the technology reference model. The bureau plans to  issue the next version of its “to be” architecture in fiscal year 2006.
In addition, the bureau reports it has developed a “high level” description of  a sequencing plan that is not yet complete; the next version of the plan is  scheduled for issuance in September 2005.
Two additional elements (one Stage 4 and one Stage 5 element) have also  been satisfied. Specifically, while EA products and processes to date have  not been independently verified and validated, the FBI hired a contractor in  April 2005 to begin performing such assessments on both the EA products  and the processes used to develop them. According to the contract  statement of work, the results of these assessments are to be shared with  the program office and reported to the steering committee.
Also, the bureau has defined a management structure and process to  formally manage EA change. According to its configuration management  plan (dated February 3, 2005), the bureau is using an automated tool to  manage critical EA work products as they are developed and changed.  Further, the bureau established a Change Management Board to resolve  critical issues, including those that require a major commitment of  resources, vary from the EA strategy, or require a policy change.
Beyond these two elements, 14 core elements in Stages 4 and 5 have yet to  be satisfied. In particular, key architecture products have yet to be  completed. As previously noted, the bureau is still in the process of  developing both its “as is” and “to be” architectures, for example. The  sequencing plan is also a work in process. (A summary of the results of our  assessment on the FBI’s satisfaction of the core elements for each of the  stages are provided in app. III.)
Discussing the bureau’s EA program, the FBI’s CIO said that significant  progress has been made, which he attributed to top-level organizational  commitment and focus on EA, as well as assignment of bureauwide IT  budget control and authority to the CIO. Despite this progress, much  remains to be accomplished before the FBI will have an effective EA  program. According to our framework, effective architecture management  is generally not achieved until an enterprise has a completed and approved  architecture that is being effectively maintained and is being used to  leverage organizational change and support investment decision making;  having these characteristics is equivalent to having satisfied all of the Stage  2 and 3 core elements and many of the Stage 4 and 5 elements.
Until the bureau gets to that stage, it will be challenged in its efforts to  implement modernized systems in a way that minimizes overlap and  duplication and maximizes integration and mission support. Our prior  reviews of federal agencies and research of architecture best practices  have shown that attempting to modernize systems without a well-defined  and verifiable architecture and associated management capabilities  increases the risk that large sums of money and much time and effort will  be invested in technology solutions that are duplicative, are not well  integrated, are unnecessarily costly to maintain and interface, and do not  effectively optimize mission performance.
section_title: Bureau Is Not Effectively Managing Its EA Contractor
paragraphs: Federal acquisition regulations and relevant IT acquisition management  guidance recognize the importance of effectively managing contractor  activities. The Federal Acquisition Regulation (FAR), for example, directs  agencies to use performance-based contracting to the maximum extent  practicable when acquiring most services. Under the FAR, performance- based contracting includes (1) defining the work to be performed in  measurable, results-oriented terms; (2) specifying performance standards  (quality and timeliness) that are tied to contractual requirements;  (3) having a quality assurance plan that describes how the contractor’s  performance in meeting requirements will be measured against standards;  and (4) establishing positive and negative contractor performance  incentives. The FAR and associated regulations also require government  oversight of contracts to ensure that the contractor (the service provider)  performs the requirements of the contract, and the government (the service  receiver or customer) receives the service as intended. However, the  regulations do not prescribe specific methods for this oversight.
Other acquisition management guidance identifies effective contractor  tracking and oversight as a key activity and describes a number of practices  associated with this activity, including   establishing a written policy for contract tracking and oversight,  designating responsibility for contract tracking and oversight activities,  establishing a group that is responsible for managing contract tracking  and oversight activities, and  using approved contractor planning documents as a basis for tracking  and overseeing the contractor.
The FBI’s approach to managing its EA contract does not include most of  the performance-based contracting features described in the FAR.  Specifically, although the contract’s statement of work defines when  products are due (i.e., timeliness standards), it does not specify the  products in results-oriented, measurable terms. For example, the statement  of work defines requirements in terms of general product descriptions such  as “as is” and “to be” architectures and a sequencing plan. Further, it does  not specify quality standards for products and does not define incentives  for addressing either timeliness or quality standards.
The bureau also does not have plans for assuring the quality of the  contractor’s work. Instead, bureau officials told us that they follow the  bureau’s long-standing approach of working with the contractor to  determine whether each deliverable is acceptable. As an example, the  bureau received a draft of its “as is” architecture on August 22, 2004.  According to bureau officials, the draft was of poor quality, and the bureau  did not accept it. The bureau then worked with the contractor to improve  the quality of the product, and after several iterations, the bureau accepted  a draft of the “as is” architecture on September 30, 2004. However, because  the bureau did not have either quality standards or a quality assurance plan,  the basis for acceptance was not available for us to independently assess.
In tracking and overseeing its contractor, the FBI also has not employed  the kind of effective practices specified in relevant acquisition management  guidance. For example, the bureau does not have a written policy to govern  its tracking and oversight activities, has not designated responsibility or  established a group for performing contract tracking and oversight  activities, and has not developed an approved contractor monitoring plan.  Instead, the bureau holds weekly status meetings with its EA contractor to  discuss progress and plans, and it is receiving incremental drafts of work  products in an effort to increase visibility into contractor activities and  thereby minimize the number of unacceptable deliverables and associated  rework.
FBI officials from the offices of the Chief Financial Officer and CIO  attributed the current contract management approach to several factors.  First, they said that the FBI has historically been challenged in developing  statements of work that clearly define requirements and establish  performance (quality and timeliness) standards, which are essential to  effective performance-based contracting. Second, these officials stated that  they are still working to define effective contract management controls.  Specifically, as part of the CIO office’s transformation, including  implementing its recently assigned agencywide authority and control over  IT resources, these officials are developing standard policies and  procedures for managing IT. In particular, these policies and procedures  are to include an FBI-wide standard life-cycle management directive that is  to define procedures for the use of performance-based contracting  methods and the establishment of tracking and oversight structures,  policies, and processes. The officials told us they began implementing parts  of the directive in late June 2005, but added that certain key practices, such  as acquisition management, were early drafts and required further  development. However, the officials were unable to provide a date for  when the drafts would be finalized and implementation of the practices  would begin.
In the absence of performance-based contracting and effective tracking  and oversight, the bureau’s ability to effectively manage its EA contractor is  constrained. This means that the FBI is at risk of taking more time and  spending more money than necessary to produce a well-defined  architecture.
section_title: Conclusions
paragraphs: Having a well-defined and enforced architecture is critical to the FBI’s  ability to effectively and efficiently modernize its mission operations and  supporting IT environment. The bureau has taken steps aimed at  developing such an architecture and has made important progress in doing  so; however, much remains to be accomplished before it will have  implemented our prior recommendations and established an effective EA  program. As it moves forward, it is important for the bureau to employ all  the effective architecture management practices that we have previously  recommended, and to do so expeditiously. Moreover, given that the FBI’s  program is heavily relying on contractor support, it is also important for the  bureau to ensure that it employs effective contract management controls  that will enable it to, among other things, define contractor work to be  performed in measurable, results-oriented terms; establish positive and  negative contractor performance incentives; and define and implement  contractor tracking and oversight processes consistent with acquisition  management guidance. Currently, the FBI does not have such controls in  place, and as a result, it is increasing the risk that it will take more time and  money to develop a well-defined EA than is necessary. If the bureau does  not begin employing the kind of effective contract management controls  contained in federal regulations and related guidance, its architecture  efforts will continue to be at risk. In turn, its systems modernization will  continue to be challenged in its ability to efficiently and effectively support  mission operations through modern IT systems.
section_title: Recommendations for Executive Action
paragraphs: In light of our prior comprehensive set of recommendations for  strengthening the FBI’s EA program, we are not making additional  recommendations at this time relative to satisfying the practices embodied  in our architecture management maturity framework.
Given the FBI’s heavy reliance on contractor assistance in developing its  EA and the state of its contract management controls, we recommend that  the FBI Director direct the Chief Financial Officer, in conjunction with the  CIO, to ensure that to the maximum extent practicable, performance-based  contracting activities, along with effective contract tracking and oversight  practices, are employed prospectively on all EA contract actions. This  should include, among other things, defining contractor work in  measurable, results-oriented terms; establishing positive and negative  contractor performance incentives; and defining and implementing  contractor tracking and oversight processes consistent with acquisition  management guidance.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the CIO and  reprinted in appendix IV, the FBI agreed that the bureau had made progress  in developing its architecture. The FBI also stated that it appreciated our  assessment and feedback on its EA program and that the bureau would  continue to strive to develop a robust EA program supported by effective  contract management practices. In this regard, the FBI cited steps under  way to strengthen its EA management foundation. The FBI also noted our  recommendation regarding the use of performance-based contracting,  stating that its use of fixed-price contracting for EA support has been  successful. We believe the FBI can benefit from increased use of  performance-based contracting techniques even under firm, fixed-priced  contracts. In this regard, the FBI agreed, stating that our recommendations  provide for effective EA contract management practices and that it is now  taking steps to increase its use of performance-based contracting. The FBI  stated that it is in the process of increasing employee awareness and  providing training on the performance-based approach.
We are sending copies of this report to the Chairmen and Ranking Minority  Members of the Senate and House Appropriations Committees. We are also  sending copies to the Attorney General; the Director, FBI; the Director,  OMB; and other interested parties. This report will also be available at no  charge on our Web site at http://www.gao.gov.
Should you have any questions about matters discussed in this report,  please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO contacts and staff who made major  contributions to this report are listed in appendix V.
section_title: Objectives, Scope, and Methodology
paragraphs: As specified in the conference report accompanying the Consolidated  Appropriations Act, 2005, our objectives were to determine (1) whether  the Federal Bureau of Investigation (FBI) is managing its enterprise  architecture (EA) program in accordance with established best practices  and (2) what approach the bureau is following to track and oversee its EA  contractor, including the use of effective contractual controls.
For the first objective, we reviewed our EA management maturity  framework, Version 1.1, which organizes architecture management best  practices into five stages of maturity. This framework is based on A  Practical Guide to Federal Enterprise Architecture, published by the  federal Chief Information Officer (CIO) Council. We compared our  framework with the ongoing efforts of the FBI’s EA program. Specifically,  we analyzed the bureau’s EA plans and products, including program  management and other plans, key architecture principles, work breakdown  structures and corresponding milestones, Enterprise Architecture Board  charters and meeting minutes, repository strategy, and EA status reports.  We also analyzed relevant policies and procedures, including the bureau’s  EA Policy and the Information Technology Life Cycle Management  Directive. Moreover, we reviewed draft architecture work products,  including iterations of the “as is” and “to be” architectures; we did not,  however, assess the contents or quality of these architectural work  products because they were in varying degrees of completion and subject  to ongoing change. Next, we compared our analyses with the EA  management maturity framework practices to determine the extent to  which the FBI was employing such effective management practices. We  also interviewed bureau officials, such as the CIO, the chief architect, and  the head of the EA program office.
For the second objective, we first reviewed key federal regulations and  best practices and guidance. In particular, we reviewed relevant federal  acquisition regulations on effective contract management, including  performance-based contracting methods. Additionally, we reviewed the  Software Engineering Institute’s Software Acquisition Capability Maturity  Model, version 1.02, for key contractor tracking and oversight best  practices. We then analyzed EA contract documentation, including task  orders, statements of work, and contract modifications. We also  interviewed FBI officials, including the contracting office’s technical  representative for overseeing the EA contractor, the chief architect, and the  head of the EA program office. We interviewed these officials to verify and  clarify our understanding of the bureau’s architecture contract  management procedures and to determine whether the bureau is  employing effective contractual controls. Additionally, we discussed with  these officials the cause and impact of the current state of the bureau’s  contract management activities and policies.
We performed our work at FBI headquarters in Washington, D.C., from  September 2004 to July 2005, in accordance with generally accepted  government auditing standards.
section_title: Detailed Descriptions of Elements in GAO’s EA Management Maturity Framework
paragraphs: Because the task of developing, maintaining, and implementing an EA is an  important, complex, and difficult endeavor, doing so effectively and  efficiently requires that rigorous, disciplined management practices be  adopted. Such practices form the basis of our EA management maturity  framework, which specifies by stages the key architecture management  structures, processes, and controls that are embodied in federal guidance  and best practices. The five stages and their associated core elements are  described below.
At Stage 1, organizations are becoming aware of the value of an EA, but  have not yet established the management foundation needed to develop  one. Stage 1 has no core elements: by default, an organization that does not  satisfy Stage 2 core elements is at Stage 1.
For Stage 2, our framework specifies nine key practices or core elements  that are necessary to provide the management foundation for successfully  launching and sustaining an architecture effort:   Ensure that adequate resources exist. An organization should have the  resources (funding, people, tools, and technology) to establish and  effectively manage its architecture. This includes identifying and  securing adequate funding to support EA activities; hiring and retaining  the right people with the proper knowledge, skills, and abilities to plan  and execute the EA program; and selecting and acquiring the right tools  and technology to support EA activities.
Establish a committee or group representing the enterprise that is  responsible for directing, overseeing, or approving the EA. This  committee should include executive-level representatives from each  line of business, and these representatives should have the authority to  commit resources and enforce decisions within their respective  organizational units. By establishing this enterprisewide responsibility  and accountability, the agency demonstrates its commitment to building  the management foundation and obtaining buy-in from across the  organization.
Establish a program office that is responsible for EA development and  maintenance. This organizational unit should be devoted to the EA  program and responsible for developing a management plan and  executing the plan. The plan should include a detailed work breakdown  structure; resource estimates (e.g., funding, staffing, and training);  performance measures; and management controls for developing and  maintaining the architecture.
Appoint a chief architect. The chief architect should be responsible and  accountable for the EA, supported by the architecture program office,  and overseen by the architecture steering committee. The chief  architect (in collaboration with the CIO, the architecture steering  committee, and the organizational head) is instrumental in obtaining  organizational buy-in for the architecture, including support from the  business units, as well as in securing resources to support architecture  management functions such as risk management, configuration  management, quality assurance, and security management.
Use a framework, methodology, and automated tool to develop the  architecture. The framework provides a formal structure for  representing the EA, while the methodology is the common set of  procedures that the enterprise is to follow in developing the  architecture products. The automated tool serves as a repository where  architectural products are captured, stored, and maintained.
Develop an architecture program management plan. This plan  specifies how and when the architecture is to be developed. It includes a  detailed work breakdown structure; resource estimates (e.g., funding,  staffing, and training); performance measures; and management  controls for developing and maintaining the architecture. The plan  demonstrates the organization’s commitment to managing architecture  development and maintenance as a formal program.
Ensure that EA plans call for describing both the “as is” and “to be”  environments in terms of business, performance, information/data,  application/service, and technology. An organization’s program  management plan should provide for defining and normalizing the  current and future architectures in terms relevant to stakeholders from  varying organization levels and disciplines.
Ensure that EA plans address security at each layer. Plans should  define how the organization will address security as a distinct area of  operational and technology emphasis within the context of each layer.
Ensure that EA plans call for developing metrics for measuring EA  progress, quality, compliance, and return on investment. Plans should  provide for developing metrics and should describe how these will be  used to measure (1) progress towards EA goals, (2) the quality of  architecture products and management processes, (3) compliance with  the architecture, and (4) EA return on investment.
At Stage 3, our framework specifies six core elements that are necessary to  focus on architecture development activities:  Issue a written and approved organization policy for EA development.  A policy defines the scope of the architecture, including the requirement  for a description of the current and target architectures, as well as an  investment road map or sequencing plan specifying the move between  the two.
Ensure that EA products are under configuration management. This  involves ensuring that changes to products are identified, tracked,  monitored, documented, reported, and audited.
Ensure that EA products describe or will describe both the “as is” and  the “to be” environments, as well as a sequencing plan. Consistent with  the EA program plans discussed in Stage 2, an organization should  ensure that the EA products being developed are enterprisewide in  scope and describe both the current and future environments, as well as  a sequencing plan for moving from the current to the target  environment.
Ensure that EA plans are described or will be described for both  environments in terms of business, performance, information/data,  application/service, and technology. Products being developed or  drafted should begin to address each of the given terms of reference, or  include placeholders for later defining the enterprise in these terms.
Ensure that business, performance, information/data,  application/service, and technology descriptions address or will  address security. This involves ensuring that each EA product  (including those describing the “as is” and “to be” environments in terms  of business, performance, information/data, application/service, and  technology) explicitly describe how enterprise security is being defined  and will be implemented.
Ensure that progress against EA plans is measured and reported. To  assist in attaining stated EA program goals and objectives, an  organization should understand and disclose its progress against plans.
As EA products emerge, their content should be assessed against the  plans to ensure that expectations are being met.
At Stage 4, during which organizations focus on architecture completion  activities, organizations need to satisfy eight core elements:   Issue a written and approved organization policy for EA  maintenance. A policy promotes enterprisewide commitment to  keeping the EA up to date. It should provide for establishing a process  for architecture maintenance, including oversight and control. It should  also identify the roles, responsibilities, and relationships of key players  in the maintenance process.
Ensure that EA products and management processes undergo  independent verification and validation. This core element involves  having an independent third party—such as an internal audit function or  a contractor that is not involved with any of the architecture  development activities—verify and validate that the products were  developed in accordance with architecture processes and product  standards. Doing so provides organizations with needed assurance of  the quality of the architecture.
Ensure that EA products describe both the “as is” and the “to be”  environments, as well as a sequencing plan. Consistent with the EA  program plans discussed in Stage 2, an organization should ensure that  the EA products completely and correctly describe both the “as is” and  the “to be” environments of the enterprise and include a sequencing plan  for migrating the organization between the two environments.
Ensure that EA products for both environments are described in terms  of business, performance, information/data, application/service, and  technology. An organization’s EA products should be defined and  normalized in terms meaningful to a wide variety of stakeholders,  ranging from the organization’s chief executive officer and strategic  planners to its technology implementers and operators.
Ensure that business, performance, information/data,  application/service, and technology descriptions address security. An  organization should explicitly and consistently address security in its  business, performance, information/data, application/service, and  technology architecture products. Because security permeates every  aspect of an organization’s operations, the nature and substance of  institutionalized security requirements, controls, and standards should  be captured in the EA products.
Ensure that the organization’s chief information officer has approved  the current version of the EA. The current version of the organization’s  completed EA should be approved by the CIO.
Ensure that a committee or group representing the enterprise or the  investment review board has approved the current version of the EA.  The current version of the organization’s completed architecture should  also be approved either by the EA steering committee or by the  investment review board.
Measure and report on the quality of EA products. An organization  should ensure that the nature and content of the EA products meet  defined quality standards. This core element entails developing a set of  metrics and assessing the products against those metrics.
At Stage 5, during which the focus is on architecture maintenance and  implementation activities, organizations need to satisfy eight core  elements:   Issue a written and approved organization policy for information  technology (IT) investment compliance with the EA. A policy that  governs the implementation of the architecture should be approved by  the organization head. The EA policy should augment architecture  development and maintenance policies by providing for an institutional  EA implementation process that is aligned with the organization’s  capital planning and investment control process.
Ensure that the organization has a process to formally manage EA  change. A formal process should be defined and implemented for  introducing changes to the architecture. This process should recognize  both internally and externally prompted change, and it should provide  for continuous capture and analysis of change proposals and informed  decision making about whether to make changes.
Make the EA an integral component of the IT investment management  process. Because the road map defines the IT systems that an  organization plans to invest in as it transitions from the “as is” to the “to  be” environment, the architecture is a critical frame of reference for  making IT investment decisions. Using the architecture when making  such decisions is important because organizations should approve only  those investments that move the organization toward the “to be”  environment, as specified in the road map.
Ensure that EA products are periodically updated. An organization will  need to periodically update its EA products depending on the volume  and degree of approved changes to the EA.
Ensure that IT investments comply with EA. An organization’s IT  investments should be aligned and comply with the applicable  components of the current version of the EA, and they should not be  selected and approved under the organization’s capital planning and  investment control process unless compliance is documented by the  investment sponsor and substantiated by the architect assessment team.
Ensure that the organization head has approved the current version of  the EA. The current version of the EA should ultimately be approved by  the head of the organization.
Measure and report return on EA investment. Like any investment, the  architecture should produce a return on investment (i.e., a set of  benefits), and this return should be measured and reported in relation to  costs. Measuring return on investment is important in order to ensure  that expected benefits from the architecture are realized and to share  this information with executive decision makers, who can then take  corrective action to address deviations from expectations.
Measure and report on compliance with the EA. An organization  should define metrics, such as number of compliance waivers requested  and number granted, to track compliance. Through such measurement  and reporting, relevant trends and anomalies can be identified, and  corrective action can be taken.
section_title: Assessment of the FBI’s EA Efforts against GAO’s EA Management Maturity Framework
paragraphs: Satisfied?
Agency is aware of EA.
The FBI has acknowledged the need for an EA,  and the Director has made its development a  management priority.
Adequate resources exist.
According to FBI officials, they have identified the  financial and human capital resources needed to  effectively manage the bureau’s architecture  program. While bureau officials stated they have  adequate financial resources to fund the program,  including sufficient contractor assistance, four of  five core architect positions identified as being  needed to staff the program office have not yet  been filled.
Committee or group representing the  enterprise is responsible for directing,  overseeing, or approving EA.
The FBI has established an Enterprise Architecture  Board to direct, oversee, and approve the EA. The  board includes upper-level management from all  the operating units, including the counterterrorism,  counterintelligence, and finance divisions.  Technical representatives, such as the chief  technology officer and chief architect, also serve on  this board.
Program office responsible for EA  development and maintenance exists.
The FBI has established a program office, called  the Enterprise Architecture Unit, which is located in  the CIO’s office. The program office is responsible  for the development, implementation, and  maintenance of the EA.
Chief architect exists.
The FBI has designated a chief architect.
EA is being developed using a  framework, methodology, and  automated tool.
The FBI initially used the Federal Enterprise  Architecture Framework and has since switched to  OMB’s five Federal Enterprise Architecture  Reference Models. The bureau is using the Popkin  System Architect tool. However, the bureau does  not have a documented methodology that defines  how EA products are to be developed. Instead of a  defined methodology, the bureau is relying on a  combination of its chief architect’s knowledge and  certain documentation, such as an EA alignment  plan that describes, among other things, the  products to be developed, the order in which they  are to be developed, the relationships among  products, and analyses that are to be performed to  help identify gaps and redundancies in the  contents of these products. However, this  documentation does not include either the specific  steps or methods that explain how the content of  products is to be developed and documented.
Satisfied?
EA plans call for describing the “as is”  and “to be” environments, and a  sequencing plan.
The EA program management plan (dated October  2004) calls for the development of “as is” and “to  be” environments as well as a sequencing plan.
EA plans call for describing the  enterprise in terms of business,  performance, information/data,  application/service, and technology.
The FBI’s EA baseline report (dated March 2005)  and other plans call for the development of  business, performance, data, applications, and  technology descriptions.
EA plans call for business,  performance, information/data,  application/service, and technology  descriptions to address security.
The FBI’s EA baseline report (dated March 2005)  and other plans call for security services to be  defined for each of the descriptions.
EA plans call for developing metrics for  measuring EA progress, quality,  compliance, and return on investment.
The EA policy (dated August 2003) and program  management plan call for developing metrics to  measure progress, quality, and return on  investment.
Written and approved organization  policy exists for EA development.
The FBI has a written policy for EA development  (dated August 2003) that was approved and signed  by the CIO.
EA products are under configuration  management.
The bureau has a configuration management plan  that defines management structures and  processes for identifying, tracking, monitoring,  reporting, and auditing changes to the architecture  products. EA products, such as the program  management plan, EA principles, initial versions of  the “as is” architecture, and EA software tool, have  been identified and placed under configuration  management in accordance with the plan.
EA products describe or will describe  the enterprise’s business,  performance, information/data,  application/service, and the technology  that supports them.
The FBI is in the process of developing its “as is”  and “to be” architectures. It reports that to date, it  has issued what it describes as “high level”  versions of each, but that these versions need  additional work to be complete. The initial version  of the “to be” includes the enterprise's business,  performance, information/data, service, and  technology descriptions. The latest draft of the “as  is” also includes all of these descriptions, except  performance. According to FBI officials,  performance was omitted due to an oversight on  their part, and they intend to address performance  in the next version of the “as is” architecture.
EA products describe or will describe  the “as is” and the “to be”  environments, and a sequencing plan.
The FBI is in the process of developing its “as is”  and “to be” architectures. It reports that to date, it  has issued what it describes as “high level”  versions of each, but that these versions need  additional work to be complete. The FBI also  reports that it has developed a “high level”  description of a sequencing plan that is not yet  complete.
Satisfied?
Business, performance,  information/data, application/service,  and technology address or will address  security.
The FBI is in the process of developing its “as is”  and “to be” architectures, as described above.  These versions of its architectures include a  description of security services. According to FBI  officials, these versions are not yet complete.
Progress against EA plans is  measured and reported.
The FBI is measuring and reporting progress  against EA plans.
Written and approved organization  policy exists for EA maintenance.
The FBI does not have a written and approved  policy for EA maintenance. While the bureau has  an EA development policy, it does not address  architecture maintenance, nor does it assign  responsibility and accountability for maintenance.
EA products and management  processes undergo independent  verification and validation.
While EA products and processes to date have not  been independently verified and validated, the FBI  hired a contractor in April 2005 to begin performing  such assessments on both the EA products and  the processes used to develop them.
EA products describe the enterprise’s  business, performance,  information/data, application/service,  and the technology that supports them.
Initial EA products describe the enterprise’s  business, performance, information/data,  application/service, and the technology that  supports them. However, the FBI reports that these  products are not completed.
EA products describe the “as is” and  the “to be” environments, and a  transitioning (sequencing) plan.
Initial EA products describe the “as is” and the “to  be” environments and a sequencing plan. However,  the FBI reports that these products are not  completed.
Business, performance, data,  application, and technology  descriptions address security.
Initial EA products include business, performance,  information/data, application/service, and the  technology descriptions that address security.  However, the FBI reports that these products are  not completed.
Organization’s chief information officer  has approved current version of EA.
The FBI is in the process of completing its EA, and  when completed, the CIO plans to approve it.
Committee or group representing the  enterprise or the investment review  board has approved current version of  EA.
The FBI is in the process of completing its EA, and  when completed, the Enterprise Architecture Board  plans to approve it.
Quality of EA products is measured  and reported.
Although the FBI is in the process of completing its  EA products, it is not currently measuring and  reporting quality. FBI plans call for the bureau to  begin measuring and reporting EA product quality  starting in fiscal year 2006.
Written and approved policy exists for  IT investment compliance with EA.
The FBI does not have a written and approved  policy addressing IT investment compliance with  EA.
Satisfied?
Process exists to formally manage EA  change.
The FBI configuration management plan defines a  process to formally manage EA change. To  manage the process, the bureau established a  change management board in January 2003. The  board reviews and determines whether to approve  changes to the current FBI environment.
EA is integral component of IT  investment management process.
The FBI is in the process of completing its EA, and  thus, it is not yet an integral part of the bureau’s IT  investment process.
The FBI is in the process of completing its EA, and  when it is complete, bureau plans call for the  products to be periodically updated.
IT investments comply with EA.
All IT investments are not evaluated for compliance  with a completed EA.
Organization head has approved  current version of EA.
The FBI does not yet have a completed EA for the  Director to approve.
Return on EA investment is measured  and reported.
The FBI is not yet measuring and reporting return  on investment.
Compliance with EA is measured and  reported.
The FBI is not yet measuring and reporting EA  compliance.
To achieve a particular stage includes satisfying the specified elements in the stage plus all elements  from previous stages. For example, to achieve Stage 3 requires achieving the Stage 3-specific  elements plus those in Stages 1 and 2.
section_title: Comments from the Federal Bureau of Investigation
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, the following people made key  contributions to this report: Gary Mountjoy, Assistant Director; Barbara  Collier; Lori Martinez; Teresa Neven; and William Wadsworth.