Information Technology: State Department Led Overseas Modernization Program Faces Management Challenges
Released: 2001-11-16T00:00:00

Letter

Background

In response to the tragic 1998 bombings in Kenya and Tanzania, the former Secretary of State established an independent panel, the Overseas Presence Advisory Panel (Panel), to analyze our overseas presence and develop recommendations for organizing and managing our embassies and consulates overseas. After visiting several overseas posts, the Panel concluded that the condition of U.S. posts and missions abroad was unacceptable and recommended eight major types of changes, which, according to its report, would improve the efficiency and effectiveness of our overseas presence. Specifically, the panel directed the President, the Secretary of State, and congressional leaders to form a partnership to implement the following eight recommendations: improve security and establish accountability for security at our overseas facilities; create the right size and shape for our overseas presence, ensuring that posts are not over- or under-staffed; ensure that U.S. facilities are quickly and efficiently renovated and are properly maintained and managed on an ongoing basis; adopt best practices for human resources management, such as improving recruitment and expanding training and promotion opportunities; invest in up-to-date information technology, including providing Internet access and e-mail to all staff; improve and reinforce consular services, such as same-day processing of visas; enact cost- and time-saving administrative reforms, such as consolidating some functions (e.g., accounting practices) and automating others; and reinforce the responsibilities and authorities of Ambassadors, who represent the interests of the President of the United States.

Because State is the lead agency for foreign affairs, President Clinton directed the former Secretary of State, on February 10, 2000, to lead a cabinet-level committee to implement the Panel’s recommendations. As part of the former Secretary’s response, the Under Secretary for Management formed an interagency Overseas Presence Committee (the Committee) in February 2000 to address three of the eight recommendations. The Committee then established three subcommittees to implement the Panel’s recommendations: Overseas Facilities, Interagency Rightsizing, and Interagency Technology. (See fig. 1 for the Committee’s structure and table 1 for allocation of responsibilities.)

In October 2000, the then Secretary of State reported on the progress made in addressing the Panel’s recommendations for seven of the eight recommendations. In response to the recommendation to invest in up-to- date information technology, State reported that it had established an acquisition and deployment strategy, preliminary system requirements, a high-level implementation plan, and an architectural concept for a common overseas knowledge management system infrastructure; established September 2000 as the goal for completing a formal program management plan, with a goal of revising the plan by April 2001 after completing evaluation of a prototype system; and established December 2001 as the goal for completing system pilot testing at two posts.

On August 8, 2001, State issued its final status report on its efforts to address the Panel’s recommended changes to the Chairman of the Senate Committee on Foreign Relations. With regard to the recommendation to upgrade information technology, State reported that the interagency group has developed and is testing a prototype of the common overseas system and that a field test of the system’s infrastructure is being conducted at pilot posts in Mexico and India. In addition, State officials told us that the date for completing system prototyping has been delayed by 13 months to May 2002, and the date for completing pilot testing at two posts has been delayed by 9 months to September 2002. Also, the three vendors selected to develop system prototypes are not scheduled to deliver these systems to State for evaluation until November 2001.

Common Overseas System Modernization: A Brief Description

The modernization program is intended to put in place a common overseas knowledge management system to facilitate unclassified information/knowledge sharing among foreign affairs agencies. Key system capabilities are to include, among other things, interagency e-mail, post-specific news and information web-based links, crisis coordination support, policy formulation support, and various administrative functions (e.g., accounting, contract management, training, and travel). According to State’s draft program management plan, the department will acquire and deploy system functionality in each of these areas in three increments, or three prioritized groupings of user requirements, which State calls pilot, full-scale implementation, and deferred. (See table 2 for planned system functionality by increment.)

The system is to use commercially available, Internet-based technology to enable users to access the system from any location using their respective agencies’ existing networks and operating platforms. The desktop Internet browser is intended to be the connection to the common overseas system and its knowledge management tools. A common website is to serve as the portal to the customized database applications, files, and communication tools to be shared across the foreign affairs community’s diverse, distributed environment. (See fig. 2 for a simplified diagram of the system.)

In 1999, the Panel estimated the cost to acquire and deploy this unclassified system at about $200 million, assuming the use of commercially available technology. About a year later, State estimated that the investment in the system through fiscal year 2004 would be about $271 million. State plans to update this cost estimate. According to State’s draft program management plan, the estimated cost to perform system prototyping and pilot testing is $18.2 million.

The conference report accompanying the Departments of Commerce, Justice, and State, the Judiciary, and Related Agencies Appropriations Act, 2001, stated the conference committee’s expectation that State would spend $17 million to carry the program through pilot testing. State has met with subcommittee staff to keep them informed of its plans and progress, and State has agreed to submit a spending plan for the pilot project, as directed by the conference report. Prototyping activities are currently scheduled for completion in May 2002.

Recent Review of Common Overseas System Modernization Program Noted Management Challenges

In June 2000, we testified that State was in the early stages of planning for the overseas-presence system modernization—establishing preliminary milestones, developing rough cost estimates, and formulating a program management plan. At that time, we expressed concern regarding (1) State’s ability to obtain agreement internally as well as from the foreign affairs partner agencies in devising and deploying a common technology solution and (2) State’s lack of an approved architecture to guide and constrain its own IT investments. In our testimony, we stated, among other things, that it was important for State to (1) carefully plan the initiative establishing realistic goals and milestones, (2) install the needed management and oversight accountability to properly guide the acquisition of the system, and (3) anticipate the steps needed to gain the full cooperation of its agency partners.

State’s Near-Term Plans Are Focused on Risk Reduction Activities

Office of Management and Budget (OMB) guidance outlines risk reduction principles that are consistent with best commercial practices and, if effectively implemented, can increase the probability that system modernization programs will succeed. Among other things, this guidance supports (1) prototyping system components before acquiring or developing the operational system, (2) using commercially available technology, and (3) securing system stakeholders’ involvement and agreement on products such as management plans and system requirements.

State’s near-term plans generally provide for each of these areas of guidance. In particular, State is currently focused on acquiring and evaluating system prototypes that are to employ either commercially available and/or government-used hardware and software products. On July 5, 2001, State selected three vendors, each of which was to develop a prototype system by October 2001. However, according to a State official, because of security and infrastructure concerns identified during testing that the vendors must address, the systems are now scheduled for delivery in November 2001. In addition, State planned to evaluate the delivered prototype systems using evaluation criteria that were to be developed by a private firm by September 28, 2001. While an initial draft of the criteria was provided on that date, according to a State official, revisions were needed, and the deadline for finalizing the evaluation criteria was changed to mid-November 2001. Although these events will delay the deadline for completing the prototype evaluation one month to May 2002, piloting activities will not be affected and will begin as scheduled. Once the prototype system solutions have been evaluated, State and its agency partners plan to use the best capabilities from each prototype to refine user requirements and develop a specification for the foundational system capabilities that will be pilot tested. State plans to pilot test these foundational capabilities in India and Mexico between April and September 2002. According to State, no commitments have been made to proceed beyond this pilot test, and any decisions about how best to proceed will have to involve agency partners, OMB, and relevant congressional committees.

In addition to prototyping the system and using commercially available products, State has also taken steps to engage other foreign affairs agencies in its prototyping activities, including securing stakeholder involvement and buy-in. Specifically, responses from the seven modernization program partner agencies that responded to our survey showed that they were generally satisfied with State’s efforts to engage them in near-term program activities. Additionally, four of these agencies stated that State had informally obtained their concurrence on such items as program plans and system requirements. As for the other three agencies, one stated that while it had reviewed some of the program documentation, it had not participated in an approval process; another stated that it had reviewed program documentation for informational purposes only and was unaware of any approval process; and the third stated that it had not reviewed any of the program documentation.

Nevertheless, we found that these and other interagency meetings, such as program working group meetings, have occurred and provided forums for informally engaging agency partners. In general, agency partners attended these meetings and participated in the program. For example, two agency partners have assigned staff to the FASI Program Office.

Available documentation and State CIO officials’ statements confirm that agency partners’ buy-in to program decisions, such as agreement on the requirements for the prototype and approval of plans, is being accomplished informally. According to State, representatives for the partner agencies agreed that their lack of opposition or comments equated to concurrence on actions, plans, and/or documents, thereby eliminating the need to obtain actual signatures to indicate approval. Although State could not provide us with any written agreements to this effect, revised minutes of an Interagency Technology Subcommittee meeting reflected this informal agreement.

Much Remains to be Done Before Acquiring the Operational System

The Clinger-Cohen Act of 1996, in combination with OMB guidance implementing the act, provide federal agencies with a framework for effectively managing modernization efforts that are consistent with best commercial practices. Among other things, OMB requires (1) a clear definition of program management roles, including an explicit definition of responsibilities and accountabilities for such things as funding, investment decisionmaking, requirements management, architecture definition and integration with legacy systems, and testing; (2) use of incremental investment management principles to minimize the enormous risk inherent in large modernization programs that involve many things to be done over many years; (3) the development, maintenance, and implementation of an enterprise architecture, which is an institutional blueprint for guiding operational and technological modernization; and (4) the use of effective IT project management processes, such as processes for managing software-intensive system acquisition projects.

For its overseas knowledge management system modernization program, State program officials told us that they plan to establish and implement each of these management controls. However, much remains to be accomplished because thus far State has devoted its attention to system prototyping activities, and, according to State officials, they first had to brief congressional staff earlier in fiscal year 2001 on the use of appropriated funds before establishing these management controls.

It is essential that these controls be effectively implemented on the overseas-presence system modernization program before State proceeds with its acquisition of the operational system. To do less would unnecessarily increase the risk that needed system capabilities will not be delivered on time and within budget.

Clear Definition of Stakeholders’ Accountability Is Critical to Proceeding Beyond Pilot Testing

OMB guidance recognizes the importance of fully and explicitly defining and ensuring understanding of program stakeholders’ areas of responsibility and accountability in managing and funding a program. Similarly, the Software Engineering Institute (SEI), recognized for its expertise in managing software-intensive systems, emphasizes the importance of clearly designating roles and responsibilities on system acquisition programs. While doing so is essential on any program, it is particularly critical for programs like the overseas-presence knowledge management modernization program because its success depends on the effective interplay of multiple federal agencies. Without well-defined, understood, and agreed-to roles in such areas as funding and acquisition management (e.g., investment decisionmaking, requirements management, architecture definition and integration with legacy systems, and testing), State cannot, for example, be assured that necessary resources will be provided (e.g., financial, human capital, and technological) and critical tasks executed.

According to State, the roles, responsibilities, and accountabilities of foreign affairs agencies have been formally defined. However, State did not provide us with documentation to support this position, as we requested. Furthermore, agency partners that responded to our survey stated that their respective roles, responsibilities, and accountabilities for this initiative have not yet been formally defined.

According to State, as the principal foreign affairs agency, its role is to provide administrative and management leadership and support for near- term prototype and pilot-testing activities, and it is the agent for bringing the foreign affairs agencies together. However, State said that it is not responsible for engaging agency partners as part of the near-term prototype and pilot-testing activities.

As noted earlier, given the stated purpose and scope of these near-term activities and State’s leadership role, this definition of roles is not currently a significant issue. However, it is essential that State and its agency partners fully define agreed-to roles, responsibilities, and accountabilities for program funding and management before moving beyond planned near-term activities. To do less would unnecessarily put the modernization program at risk.

Incremental Investment Management Could Reduce Overall Program Risks

Incremental investment management involves three fundamental components: (1) developing/acquiring a large system in a series of smaller projects or system increments; (2) individually justifying investment in each increment on the basis of costs, benefits, and risks; and (3) monitoring actual benefits achieved and costs incurred on ongoing increments and modifying subsequent increments/investments to reflect lessons learned. Using this approach, agencies can avoid discovering too late that their systems are not cost beneficial and reduce the risks associated with large, expensive projects. This approach allows overlap and smooth transition among increments because lessons learned from actual results of ongoing increments would be monitored and evaluated continuously so that these results will be available for use in defining and justifying future increments.

The Clinger-Cohen Act of 1996 requires agencies to follow, to the maximum extent practical, an incremental approach to investing in IT projects. Also, OMB policy requires that investments in major systems be implemented in increments, with each increment delivering measurable benefits. More specifically, OMB’s Capital Programming Guide describes the use of modular contracting, or incremental investment, including its application and benefits. In particular, OMB states that project increments should provide for the following: Separability: Each increment should be fully funded, have substantial programmatic use that is not dependent on any subsequent increment, and be capable of performing its principal functions, even if no subsequent increments are acquired. Interoperability: Each increment should comply with a common architecture or commercially acceptable technology standards and should be compatible with and capable of being integrated with other increments.

Performance requirements: Each increment’s performance requirement should be consistent with the performance requirements of the completed overall system and should address interface requirements with other increments.

In short, incremental investment helps to mitigate the risks inherent in acquiring/developing large systems by dividing a single, large program into smaller, independently useful components with known and defined relationships and dependencies. It is well understood that making investment decisions up front on large-scale, long-term programs is risky because their economic justification is based on costs, benefits, and risks that are difficult to forecast reliably, partially because later increments are not well understood or defined and partially because they are subject to change in light of experience on nearer term increments and changing business needs. Through incremental investment management, organizations can reduce the level of program risk and complexity faced at any one time by spreading this risk and complexity across a series of smaller investments; permit the delivery of some part of the expected business value earlier rather than waiting until later for the total, but more uncertain, business value; continuously monitor and evaluate the delivery of cost and benefit expectations on ongoing increments and use this information to better define and economically justify these increments; and permit later increments to exploit technology advances or accommodate evolving business needs.

According to modernization officials, preliminary plans provide for acquiring and investing in the overseas knowledge management system in three increments: pilot, full-scale implementation, and deferred. However, State has not yet established specific plans and management processes for acquiring the operational system because, according to State’s CIO, it does not have congressional authorization to do so. We were unable to corroborate State’s position as to why it has not established these plans and processes.

In addition, modernization officials stated that the ongoing system prototyping and planned pilot-testing activities were not economically justified as a separate increment. However, these officials noted that all future incremental investments in the system will be economically justified, and the results and lessons learned from prior increments will be used to define and make investment decisions about future increment(s). Additionally, they stated that future increments will also be evaluated to ensure that return-on-investment commitments are met.

It is critical for State to follow through on its stated commitments to implement incremental investment practices. Otherwise, it risks making a single, monolithic investment decision for the entire program. This “all or nothing” approach to investing in IT has historically resulted in agencies investing huge sums of money in systems that do not provide commensurate benefits, and thus has been abandoned by successful organizations. The need to avoid this pitfall was a major impetus for the investment management reforms of the Clinger-Cohen Act.

Enterprise Architectures Are Essential Tools for Guiding and Constraining Investment Decisions

Enterprise architectures are essential tools for effectively and efficiently reengineering business processes and for implementing and evolving their supporting systems. Enterprise architectures systematically capture—in useful models, diagrams, and narrative—the full breadth and depth of the mission-based mode of operations for a given enterprise, which can be (1) a single organization or (2) a functional or mission area that transcends more than one organizational boundary (e.g., financial management, acquisition management, or overseas-presence foreign affairs activities).

An architecture describes the enterprise’s operations in both (1) logical terms, such as interrelated functions, information needs and flows, work locations, and system applications, and (2) technical terms, such as hardware, software, data, communications, security attributes, and performance standards. It provides these perspectives for both the enterprise’s current, or “as is,” environment and its target, or “to be,” environment; it also provides an IT capital investment road map for moving between the two environments.

The development, implementation, and maintenance of enterprise architectures are recognized hallmarks of successful public and private sector organizations. Managed properly, an enterprise architecture can clarify and help optimize the interdependencies and interrelationships among an organization’s business operations and the underlying IT infrastructure and applications that support these operations. Our experience with federal agencies has shown that attempting to define and build major IT systems without first completing an enterprise architecture often results in systems that are duplicative, not well integrated, unnecessarily costly to maintain and interface, and not effectively optimizing mission performance.

Federal CIO Council guidance defines a set of recognized key practices (management structures and processes) for developing and implementing an enterprise architecture that are employed by successful public and private sector organizations. Among other things, these practices include the following: Because the architecture is a corporate asset for systematically managing institutional change, the head or leader of the enterprise should support and sponsor the architecture, giving it a clear mandate in the form of an enterprise policy statement. Such support is crucial to gaining the commitment of all organizational components of the enterprise, all of whom should participate in developing and implementing the enterprise architecture.

The enterprise architecture effort should be directed and overseen by an executive body, empowered by the head(s) of the organization(s), with members who represent all stakeholder organizations and have the authority to commit resources and to make and enforce decisions for their respective organizations.

The enterprise architecture effort should be led by a Chief Architect who reports to the enterprise CIO, and it should be managed as a formal program. A formal program entails creating a program office; committing core staff; implementing a program management plan that details work breakdown structure and schedules; allocating resources and tools; performing basic program management functions (e.g., risk management, change control, quality assurance, configuration management); and tracking and reporting progress against measurable goals.

The enterprise architecture should conform to a specified framework.

Although State modernization officials have acknowledged the importance of having an overseas-presence enterprise architecture and using this architecture to guide and constrain investment in a common overseas knowledge management system, to date substantive progress in this area has not occurred. Currently, State and its agency partners have yet to either develop the architecture or establish effective management structures and processes for developing, maintaining, and implementing one, such as those specified in federal CIO Council guidance. More specifically, although State established an Enterprise Architecture Study Group in December 2000, the group met only twice and progressed no further than drafting a charter and setting a goal of completing the architecture by November 2001.

On June 8, 2001, State designated a Chief Architect to lead this effort. In July 2001, State reconvened the group, renaming it the Foreign Affairs Systems Integration Architectural Working Group, and set February 15, 2002, as the new goal for completing the architecture. Since reconvening, the working group has held three meetings to begin planning for development of the overseas-presence enterprise architecture and developed a draft charter. The working group has also begun surveying its agency partners to obtain general information on architectural requirements of overseas locations. State’s ongoing prototyping and planned piloting efforts, which are intended to provide information to better understand business and technology requirements and alternate approaches, should facilitate development of the architecture.

It is essential that State and its partner agencies effectively develop, maintain, and implement an enterprise architecture for its overseas modernization program. The federal CIO Council architecture management guide provides a road map for doing so. Our experience with system modernization programs in other federal agencies has shown that attempting to define and build major systems without first completing an enterprise systems architecture often results in systems that are duplicative, not well integrated, unnecessarily costly to maintain and interface, and not effectively optimizing mission performance.

Disciplined Acquisition Management Processes Can Further Reduce Program Risks

The Clinger-Cohen Act requires agency CIOs to establish effective IT management processes, such as processes for acquiring software. The SEI, recognized for its expertise in software process controls, has developed a model approach for software acquisition. This model outlines the key acquisition process management controls that, if implemented effectively, can greatly increase the chances of acquiring software- intensive systems that provide promised capabilities on time and within budget. The key processes are as follows: Software Acquisition Planning: Ensures that reasonable planning for the software acquisition is conducted and that all aspects of the total software acquisition effort are included in these plans.

Solicitation: Ensures that a request for proposal that delineates a project’s software-related requirements is prepared, and, consistent with relevant solicitation laws and regulations, that a contractor that can most cost- effectively satisfy these requirements is selected.

Requirements Development and Management: Establishes and maintains a common and unambiguous definition of software requirements among the acquisition team, the system users, and the software development contractor.

Project Office Management: Provides for management of the activities within the project office and supporting contractors to ensure a timely, efficient, and effective software acquisition.

Contract Tracking and Oversight: Ensures that the software development contractor performs according to the terms of the contract; needed contract changes are identified, negotiated, and incorporated into the contract; and contractor performance issues are identified early, when they are easier to address.

Evaluation: Determines whether the acquired software products and services satisfy contract requirements prior to acceptance.

Transition and Support: Provides for the effective and efficient “hand-off” of the acquired software products to the support organization responsible for software maintenance.

Acquisition Risk Management: Identifies risks as early as possible and adjusts the acquisition to mitigate those risks.

Within these key processes, SEI identifies key practices that are needed to effectively execute each key process. Among others, these key practices include: (1) designating responsibility for activities and assigning adequate resources; (2) having a written policy; (3) developing, documenting, and adhering to a plan; (4) performing management review activities; and (5) measuring the status of key activities, and using these measurements to make decisions.

According to State program officials, the modernization program will be managed in accordance with State’s methodology, Managing State Projects (MSP). MSP specifies major project management activities to be performed, the products to be generated, and the control gates (i.e., decision points) to be used to ensure that projects are ready for the next phase in their life cycle. We analyzed State’s MSP and found that it is consistent with best practice models, such as SEI’s acquisition model.

For its near-term prototyping and pilot-testing activities, which do not demand the level of management process control rigor and discipline associated with acquiring a system solution intended for operational use, State is employing adequate acquisition management process controls. For example, State has defined a work breakdown structure and schedule to guide near-term activities. Further, State CIO officials told us that management plans relating to key process areas have been drafted, although they did not provide us with copies of the draft plans, as we requested. According to State, these plans will be available when finalized and approved. These officials also told us that program management is using two automated tools to track the status of program activities. We verified that State has been using one of these tools to track program- related tasks.

According to State, it has not yet established acquisition management controls for the future operational version of the overseas-presence knowledge management system because it does not have congressional authorization to plan beyond the pilot project for this initiative. Beyond intentions to employ those controls already in place for key processes for the prototype and pilot efforts (i.e., designating a responsible official and having a written policy), State has important practices that will need to be established before it is ready to acquire the operational system. Table 3 summarizes the key process areas for managing acquisition of the operational version of the system that already exist and those that will need to be established.

It is imperative that State ensures that it fully implements key acquisition management process controls, as defined in its MSP and the SEI model, on its overseas knowledge-management modernization program before acquiring the operational system. Our reviews of other agency modernization efforts have shown that failure to implement rigorous and disciplined acquisition processes on a given program or project can lead to systems that do not perform as intended, cost more than expected, and are not delivered on time.

Conclusions

The scope of the near-term activities for State’s overseas-presence knowledge management modernization program, as described by State CIO officials and described in draft plans, is appropriately limited to system prototyping and associated pilot testing of a version of the operational system. In light of this scope, as well as State’s leadership role in managing and funding these near-term efforts and its efforts to date to bring its agency partners together, there is currently an adequate level of (1) explicit definition and understanding surrounding the overseas- presence agencies’ respective responsibilities and accountabilities, (2) agency involvement in the program, and (3) State’s management control discipline.

However, before State can effectively acquire and deploy the operational knowledge management system, it needs to more explicitly define stakeholder agencies’ program management and funding roles, as well as establish more rigorous and disciplined management controls. Accordingly, it is critical for State to also move aggressively in the near term to (1) clearly define, through written agreements, the respective management and funding responsibilities and accountabilities of the program’s agency partners and (2) institute key modernization management controls, including incremental investment management practices, enterprise architecture governance, and software acquisition management rigor and discipline. While State has verbally committed to addressing these areas, much remains to be done before State and its agency partners will be ready to proceed beyond prototyping and pilot testing and begin acquiring the system solution intended for operational deployment and use. Attempting to acquire the operational system solution without these controls risks not delivering needed system capabilities on time and within budget.

Recommendations for Executive Action

To provide for an explicit definition and understanding of the respective roles of all overseas-presence modernization agency stakeholders, we recommend that the Secretary of State develop and submit a proposal for OMB and/or congressional action, whichever the Secretary deems appropriate, within 60 days of the date of this report that clearly defines and assigns responsibilities and accountabilities for each overseas- presence agency regarding the knowledge management system modernization program. We further recommend that the Secretary direct the Under Secretary for Management to ensure that this proposal (1) sets expectations for each overseas-presence agency, including the State Department, relative to modernization management and funding and (2) is coordinated with and agreed to by the Under Secretary’s counterparts in each of the partner agencies.

To ensure that key system modernization management controls are implemented before the operational system is acquired, we recommend that the Under Secretary of State for Management direct State’s CIO, as chair of the Interagency Technology Subcommittee, to ensure that the subcommittee allows only architecturally compliant knowledge management system investments to be approved and funded, unless the subcommittee issues a written waiver in response to a written justification. Furthermore, we recommend that the Under Secretary direct the CIO to ensure that architecturally compliant investments in the system be made incrementally. To this end, the CIO should ensure that processes and structures are established for (1) acquiring the system in a series of smaller system increments; (2) individually justifying investment in each increment on the basis of costs, benefits, and risks; and (3) monitoring actual benefits achieved and costs incurred on ongoing increments and modifying subsequent increments to reflect lessons learned.

To further ensure that key management controls are implemented on the overseas-presence knowledge management system modernization, we recommend that the Secretary of State designate as a modernization program priority the development, implementation, and maintenance of an overseas foreign-affairs presence enterprise architecture. To this end, we recommend that the Secretary direct the Under Secretary for Management to establish an enterprise architecture steering committee, chaired by the Under Secretary and composed of the Under Secretary’s counterparts from the other overseas-presence agencies. We also recommend that this steering committee be assigned responsibility and accountability for ensuring that a complete and approved enterprise architecture is developed and available to guide and constrain overseas-presence system modernization efforts. To assist the steering committee, we recommend that the Under Secretary direct the State CIO to establish an enterprise architecture project office, headed by the recently named Chief Architect, to develop and maintain the enterprise architecture and submit appropriate versions of this architecture to the steering committee for approval. We further recommend that the State CIO, at a minimum, ensure that the project office (1) include staff from each of the overseas-presence agencies; (2) integrate the architecture with these agencies’ respective operational and systems environments; and (3) adhere, as appropriate, with the federal CIO Council’s published guidance on managing enterprise architectures.

To further address State’s need to implement modernization management controls, we recommend that the Under Secretary for Management direct the State CIO to ensure that the overseas-presence system modernization office, consistent with State’s project management requirements and recognized models for effective software acquisition, establishes and implements rigorous and disciplined acquisition processes, including processes for acquisition planning, contractor solicitation, requirements development and management, contractor tracking and oversight, testing and evaluation, transition to support the acquired system, and risk management.

Until these recommended modernization management controls are in place, we also recommend that the Secretary limit further investment in the overseas-presence knowledge management modernization program to (1) conducting ongoing prototyping and limited pilot testing and (2) establishing needed modernization management controls.

Agency Comments and Our Evaluation

In written comments on a draft of this report (reprinted in appendix II), State’s Acting Chief Financial Officer agreed with our findings and conclusions that its management controls over near-term prototype and pilot-testing activities are adequate, but disagreed with our finding and conclusion that State’s existing controls are not adequate for effectively managing the more demanding next phase of the program—acquisition and deployment of the operational system. Furthermore, State disagreed with our recommendations for instituting stronger management controls before proceeding to this next phase. According to State, our recommendations should be deleted from the report because (1) we reported that management controls over near-term activities are adequate and thus corrective actions are not needed; (2) except for clearly defining and agreeing to all agencies’ roles and responsibilities, State’s existing management controls are adequate for acquiring and deploying the operational system; and (3) the lead agency for the next phase of the program is uncertain and thus the recommendations are premature. At the same time, State said that most of the recommendations might more reasonably be submitted as suggestions for consideration to the lead agency for the next phase of the program.

We do not agree with State’s comment that its existing management controls are adequate for the next phase of the program. As stated in our report, the nature and purpose of system prototyping and pilot testing are fundamentally different; thus, the associated risks are less significant than those related to acquiring a system that is to be operationally deployed. As a result, it is widely recognized that the level of management control to effectively put in place an operational system, particularly a system like the overseas knowledge-management system that involves multiple agencies, is greater than is needed to prototype or pilot test the system. Thus, for example, while State’s lack of an enterprise architecture to guide and constrain near-term prototype and pilot-testing activities does not introduce significant program risk, it would introduce considerable risk for State to begin the next program phase without this architecture. Similarly, while State’s approach to managing its investment in near-term activities is adequate, effectively acquiring and deploying the operational system will require rigorous management processes that, for example, provide for incremental economic justification and measurement of system return-on-investment, and provide for fully implementing disciplined acquisition management practices in such areas as managing system requirements.

We also do not agree with State’s rationale for deleting the report’s recommendations because these recommendations (1) are not intended to correct problems with management of the prototype and pilot activities, as State asserts, but rather to proactively provide a recommended road map for how to successfully manage the next phase of the program; (2) are consistent with our conclusions that existing management controls are not adequate for the next phase of the program; and (3) recognize that prudent planning requires that agency roles and responsibilities for the next phase of the program, including designation of the lead agency, be clearly defined and agreed to during this phase to better ensure program continuity and to avoid unnecessary delays. Accordingly, the recommendations are both appropriate and warranted.

State’s written comments, along with our responses, are reproduced in appendix II.

We are sending copies of this report to the Chairman and Ranking Minority Members of the Senate Committee on Governmental Affairs; House Committee on Government Reform; Senate Committee on Foreign Relations; House Committee on International Relations; the Subcommittee on Commerce, Justice, State, and the Judiciary, Senate Committee on Appropriations; and the Subcommittee on Commerce, Justice, State, the Judiciary and Related Agencies, House Committee on Appropriations. We are also sending copies to the Director, Office of Management and Budget; the Secretary of State; the Under Secretary for Management; and the Chief Information Officer, the Department of State. Copies will be made available to others upon request.

If you have any questions on matters discussed in this report, please call me at (202) 512-3439 or Cynthia Jackson, Assistant Director, at (202) 512- 5086. We can also be reached by e-mail at hiter@gao.gov and jacksonc@gao.gov, respectively. Key contributors to this assignment are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) what State’s plans are for a multiagency, common platform, knowledge management system, and (2) whether State has effective controls in place for acquiring this system, that is, defined roles for State and its agency partners and effective controls over its investment, enterprise architecture, and system acquisition management processes.

To determine State’s plans for the system, we reviewed State budget submissions, relevant congressional direction (hearing testimony, transcript, and related legislation), the results of our prior review, the Overseas Presence Advisory Panel report and recommendations, and available State modernization program management plans and documents. Additionally, we interviewed State modernization program officials, including the Deputy Chief Information Officer (CIO)/Chief Knowledge Officer, Deputy Chief Knowledge Officer, and the Director for Standards, Policy, Planning, and Architecture, to discuss State’s near- and long-term plans, as well as representatives of the Overseas Presence Committee and the Interagency Technology Subcommittee to discuss the status of these plans and activities, and inter- and intra-agency coordination concerns. To confirm the extent to which State had engaged its agency partners, we surveyed State’s eight agency partners to determine whether they (1) were satisfied with State’s efforts to engage them in the program, (2) had reviewed and approved the program plans and related documents, and (3) believed that State was adequately addressing their requirements and/or information needs. In surveying State’s agency partners, we contacted and received responses from those individuals designated by their respective agencies as the authorized representatives for this initiative.

To determine whether State has effective controls in place for acquiring the system, we did the following.

To determine whether State and its agency partners’ roles, responsibilities, and accountabilities had been formally defined for each phase of the program (e.g., prototype, pilot), we reviewed documentation provided by State (e.g., briefing slides, meeting minutes, working group charters), interviewed State officials, and surveyed State’s eight agency partners, as described above.

To determine State and its agency partners’ plans for managing investment in the system, we reviewed relevant legislative requirements and associated OMB guidance, as well as applicable State policy and guidance, on incremental investment management. We then obtained and analyzed documentation outlining State’s investment review process for selecting, controlling, and evaluating projects since plans are currently for State to fully fund the project (prototype and pilot). Because documentation on State’s investment management activities to date and future plans for this system were not available, we interviewed officials from the modernization program office [Foreign Affairs Systems Integration (FASI) Office] and Deputy CIO for Architecture, Planning, and Regulations Office to determine if State and its partners planned to use an incremental investment management strategy. We then compared available information on State’s investment management plans against relevant federal requirements and guidance to identify if any variances existed.

To determine whether an enterprise architecture exists for the system to guide and constrain investment, we reviewed relevant legislative requirements and associated OMB guidance, as well as federal CIO Council published guidance and applicable State policy and guidance,on developing, maintaining, and implementing an enterprise architecture.

We then reviewed available documentation, including plans and management structures, a charter, process controls, and architectural artifacts, as well as interviewing FASI Program Office officials, including the Director for Standards, Policy, Planning, and Architecture, to determine the state of any enterprise architecture development and implementation efforts. We then compared the available information to relevant federal requirements and guidance to identify if any variances existed.

To determine whether State and its agency partners have the needed capabilities to effectively acquire the system, we obtained and analyzed documentation on agencywide policies and procedures governing system acquisition efforts, which State refers to as its Managing State Projects methodology. We then reviewed available best practice models on system and software acquisition management (e.g., the Software Engineering Institute (SEI) model) and compared the primary components of the Managing State Projects methodology against the primary components of SEI’s model to determine if any variances existed. SEI’s primary components include software acquisition planning, solicitation, requirements development and management, project office management, contractor tracking and oversight, evaluation, transition and support, and risk management. Because we found that the primary components of State’s methodology were consistent with best practices, we obtained and reviewed available program management and system documentation and interviewed FASI Program Office officials to determine the extent to which the methodology was being implemented on the program.

We performed our work at State Department headquarters in Washington, D.C., from October 2000 through November 2001 in accordance with generally accepted government auditing standards.

Appendix II: Comments From the Department of State

GAO Comments

1. See comments 2 through 12. 2. We did not address the adequacy of funding for near-term activities as part of this review. 3. We disagree. Our report clearly links our findings, conclusions, and recommendations. Specifically, the report states that the level of management rigor and discipline needed to effectively manage prototype and pilot-testing activities is not as demanding as that needed for acquisition and deployment of a system that is to be used operationally. For example, while State’s lack of an enterprise architecture to guide and constrain prototype and pilot-testing activities does not introduce significant risk, it would introduce considerable risk for State to begin the next phase of the program without this architecture. Accordingly, we concluded that State’s existing management controls were not adequate for the next phase of the program, and we made recommendations that are intended to proactively provide a framework for effectively managing the more demanding next phase of the program. We made no recommendations intended to correct problems with State’s management of its near-term activities because we concluded that controls over the less demanding prototype and pilot-testing activities were adequate. 4. We disagree. Our report clearly describes how State’s existing management controls are not adequate for the next phase of the program. For example, State does not yet have an enterprise architecture to effectively guide and constrain investment in the system. Our experience with federal agencies has shown that attempting to define and build major systems without first completing and using an enterprise architecture often results in systems that are duplicative, not well integrated, unnecessarily costly to maintain and interface, and not effectively optimizing mission performance. Similarly, State has not yet implemented an effective process for managing requirements for an operational system, as evidenced in the report by the fact that agency partners have not formally approved requirements for the prototype and pilot systems. Although not having formally approved requirements for building prototypes and conducting pilot tests does not introduce significant risks, it would introduce considerable risk for State to acquire a system that is to be operationally deployed without having formally approved requirements. 5. We agree that State has not proposed program and management plans beyond the prototype and pilot-testing activities. 6. We agree. State’s plans provide for preparing a prototype and pilot testing evaluation report. Also, such a report will be needed by not only the lead agency for acquiring and deploying the operational system, but also all participating agencies that will need to integrate the operational system with their respective legacy systems. 7. See comments 3, 4, and 5. 8. We agree. As stated in our report, the Secretary of State was directed to lead a cabinet-level committee to implement the recommendations of the Overseas Presence Advisory Panel, which included acquiring and deploying a multiagency common platform knowledge management system. While we recognize the possibility that State might not be the lead agency for the next phase of the program, we strongly believe that, to allow for program continuity and avoid unnecessary delays, these issues must be addressed now. Since State is currently the lead agency for the program, the recommendations are being directed to the Secretary of State. 9. We agree with State’s comment, and note in our report, that the roles, responsibilities, and accountabilities of State and its agency partners would need to be fully and formally defined before moving beyond planned near-term activities. Although State asserted during the course of our review, and in its comments, that its present informal structure for managing agency relationships is working and its comments assert that it provided documents to us attesting to this on numerous occasions, this documentation, while voluminous and repetitive, did not demonstrate what the current process is and whether it was working. As a result, we were forced to expand the scope of work to include a survey of State’s agency partners to determine whether State had engaged them in program- related activities.

From our survey, we were able to conclude that State had taken adequate action to engage the agencies. Nevertheless, some agencies raised concerns that must be addressed before beginning the next phase of the program. For example, one agency partner stated that it had not participated in an approval process for program planning documentation, another partner stated that it was unaware of any approval process for planning-related activities, and a third stated that it had not reviewed any of the program planning documentation. Moreover, one agency partner stated that a more formal program governance structure is needed, especially for reviewing and approving such documents as the program’s concept of operations and the system’s functional requirements. A similar statement was made in a meeting of the State-led Foreign Affairs Systems Integration Architectural Working Group. Specifically, it was stated that protocols needed to be established, such as officially listing the documents that would be shared so that all participants would be aware of these documents and have the opportunity to review them before making any decisions about how to proceed. While these issues are less significant for program near-term activities, they should be resolved before moving beyond system prototyping and piloting activities. Accordingly, our report contains proactive recommendations for doing so. 10. We agree with State’s comment that the need to formally define agency roles, responsibilities, and accountabilities should be focused on the next phase of the program. We disagree that there is a need to eliminate a reference to insufficiency in defining existing roles, responsibilities, and accountabilities regarding near-term program prototype and pilot-testing activities because no such reference exists. In fact, our report concludes, and State acknowledges in its comments, that management controls over near-term activities are adequate. 11. See comments 3 and 4. 12. See comments 3, 4, and 5.

Appendix III: GAO Contact and Staff Acknowledgment

GAO Contact

Acknowledgments

In addition to the person named above, Nabajyoti Barkakati, Katherine I. Chu-Hickman, Lester P. Diamond, Patrick R. Dugan, Sophia Harrison, and Kenneth A. Johnson made key contributions to this report.