Report to the Chairman, Committee on

International Relations, House of

Representatives

United States General Accounting Office

GAO

November 2001 INFORMATION

TECHNOLOGY

State Department

Led Overseas

Modernization

Program Faces

Management

Challenges

GAO-02-41
Page i GAO-02-41 State-Led Modernization Initiative

Letter 1

Results in Brief 2

Background 4

State’s Near-Term Plans Are Focused on Risk Reduction Activities 13

Much Remains to be Done Before Acquiring the Operational

System 15

Conclusions 25

Recommendations for Executive Action 25

Agency Comments and Our Evaluation 27

Appendix I Objectives, Scope, and Methodology 30

Appendix II Comments From the Department of State 33

Appendix III GAO Contact and Staff Acknowledgment 47

Tables

Table 1: Summary of Management Responsibilities and Functions

of the Overseas Presence Committee and its Entities 7

Table 2: Planned System Functionality by Increment 9

Table 3: Summary of Needed Process Controls for Managing the

Acquisition of the Operational Version of the System 24

Figures

Figure 1: Management Structure Established to Address Several

Panel Recommendations 6

Figure 2: Simplified Diagram of Target Overseas Knowledge

Management System 11

Contents
Page ii GAO-02-41 State-Led Modernization Initiative

Abbreviations

CIO chief information officer

DTS Department of State Telecommunications Service

FASI Foreign Affairs Systems Integration

IT information technology

ITS Interagency Technology Subcommittee

MSP Managing State Projects

OMB Office of Management and Budget

SEI Software Engineering Institute
Page 1 GAO-02-41 State-Led Modernization Initiative

November 16, 2001

The Honorable Henry J. Hyde

Chairman

Committee on International Relations

House of Representatives

Dear Mr. Chairman:

Dramatic changes continue to occur in the world’s economic, political,

technological, and environmental landscapes. Increases in global trade,

transnational terrorism and organized crime, and international health

concerns, for example, are combining to create a more complex, more

vulnerable world scene. To promote U.S. interests in the face of such rapid

change, 24 federal agencies are collectively engaged in foreign affairs

activities at 255 overseas locations in 162 countries.1

As our nation’s lead federal agency for foreign affairs, the Department of

State (State), among other things, is responsible for coordinating and

supporting federal agencies’ international activities, such as providing a

means for effective interagency information sharing. Currently, this means

does not exist, resulting in agencies not getting the right information to the

right people at the right time. To address this situation, State is leading a

multiagency program to modernize the information technology (IT)

environment supporting federal agencies’ overseas operations. In this role,

State is acting on behalf of the other foreign affairs agencies to acquire and

test a common knowledge management prototype and pilot system.2

 1

These agencies include the Departments of Agriculture, Commerce, Defense, Energy,

Health and Human Services, Interior, Justice, Labor, Transportation, Treasury, and

Veterans Affairs; the Environmental Protection Agency, the Federal Emergency

Management Agency, the General Services Administration, the National Aeronautics and

Space Administration, the Library of Congress, the Social Security Administration, the

American Battle Monuments Commission, the Agency for International Development, U.S.

Trade Representative, the Overseas Private Investment Corporation, the Trade and

Development Agency, the National Science Foundation, and the Peace Corps. These

agencies are represented overseas in 162 countries at 255 posts (162 embassies, 76

consulates, 17 other missions and offices).

2

Knowledge management involves the use of business processes and intellectual and

technological assets to promote and provide for collaboration and information exchange

for the purpose of accomplishing mission goals and objectives.

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-02-41 State-Led Modernization Initiative

This report provides our assessment of State’s efforts to lead this program

to modernize the IT environment supporting federal agencies’ overseas

operations. As agreed with your office, our objectives were to determine

(1) what State’s plans are for acquiring a multiagency, common-platform

knowledge management system and (2) whether State has effective

controls in place for acquiring this system, that is, defined roles for State

and its agency partners3

 and effective controls over its investment,

enterprise architecture, and system acquisition management processes. In

conducting this review, we compared State’s current and planned

activities against legislative requirements, federal guidance, and best

practices relevant to systems modernization. Our objectives, scope, and

methodology are presented in more detail in appendix I.

State is in the early, formative phase of a long-term plan to acquire and

deploy a common knowledge management system for overseas-based

agencies engaged in foreign affairs activities. This system is to provide

functionality ranging from basic Internet access and e-mail to missioncritical policy formulation and crisis management support. While detailed

long-term modernization plans understandably do not yet exist, near-term

plans do. These plans show that State will focus first on prototyping and

pilot testing a system to better understand user requirements and

alternative design options. The plans also show that State has assumed

responsibility and accountability for funding and leading the management

and administration of these near-term activities. System prototyping and

pilot testing, which involve investing a relatively small amount of time and

resources in building and evaluating a much simpler version of the

operational system solution, can be effective risk-reduction measures for

large system modernization programs. State currently plans to complete

prototype evaluation by May 2002 and pilot testing by September 2002.

To lead the near-term activities, State is employing informal management

controls, which are adequate given the department’s stated purpose and

scope of these activities. However, acquiring and deploying system

capabilities that are intended for operational use, particularly a system like

this that involves multiple agencies and performs mission-critical

 3

State’s primary agency partners include the U.S. Agency for International Development,

the Peace Corps, and the Departments of Defense, Justice, Treasury, Agriculture,

Transportation, and Commerce. Together, State and these eight agencies represent 99

percent of our overseas presence. State and the Department of Defense represent about 80

percent of our foreign overseas presence.

Results in Brief
Page 3 GAO-02-41 State-Led Modernization Initiative

functions, requires a much greater level of management discipline than

that needed for system prototyping and pilot testing. Specifically, the more

rigorous management controls that are needed include (1) explicitly

defining a multiagency governance and funding structure for acquiring and

deploying the operational system; (2) ensuring that time and resources are

invested in economically justified, architecturally compliant increments of

the operational system and verifying that increments meet return-oninvestment expectations before investing heavily in later system

increments; (3) having and using an overseas-presence enterprise

architecture as a blueprint to guide and constrain system investments; and

(4) having and following mature and disciplined software acquisition

management capabilities. To date, it is appropriate that State has not yet

established these controls because its focus has been on evaluating system

prototypes. However, without these more rigorous controls, it is unlikely

that State and its agency partners will deliver needed operational system

capabilities on time and within budget.

During the latter stages of our review, State initiated steps to establish

some of these management controls and committed to establishing others.

However, much remains to be accomplished, and plans do not yet exist for

establishing some controls. Accordingly, we are making recommendations

to the Secretary of State, as the designated lead official for the overseaspresence system modernization program, that are intended to ensure that

each of these control areas is fully addressed before acquisition of the

operational system solution begins.

In commenting on a draft of this report, State agreed with our findings and

conclusions that its management controls over near-term prototype and

pilot-testing activities are adequate, but disagreed that the department’s

controls for managing the next phase of the program are not adequate.

Additionally, State commented that our recommendations should be

deleted because (1) we reported that management controls over near-term

activities are adequate and thus corrective actions are not needed; (2)

except for defining agency roles and responsibilities, State’s existing

management controls are adequate for the next phase of the program; and

(3) the lead agency for the next phase of the program is uncertain and thus

the recommendations are premature.

We disagree with State’s comments. First, the level of management rigor

and discipline needed to effectively manage prototype and pilot-testing

activities, as discussed in the report, is not as demanding as that needed

for acquisition and deployment of a system that is to be used

operationally. Thus, for example, while State’s lack of an enterprise
Page 4 GAO-02-41 State-Led Modernization Initiative

architecture to guide and constrain prototype and pilot-testing activities

does not introduce significant program risk, it would introduce

considerable risk for State to begin the next phase of the program without

this architecture. Second, our recommendations are appropriate and

warranted because (1) they are not intended to correct problems with

management of the prototype and pilot activities, but rather to proactively

provide a recommended road map for how to successfully manage the

next phase of the program; (2) the existing controls, as explained above,

are not adequate for the next phase of the program; and (3) even though

the possibility exists that State might not be the lead agency for the next

phase of the program, prudent planning dictates that the agencies’ roles

and responsibilities for the next phase be clearly defined and agreed to

sooner rather than later to provide continuity to the program and avoid

unnecessary delays.

State’s written comments, along with our responses, are reproduced in

appendix II.

In response to the tragic 1998 bombings in Kenya and Tanzania, the former

Secretary of State established an independent panel, the Overseas

Presence Advisory Panel (Panel), to analyze our overseas presence and

develop recommendations for organizing and managing our embassies and

consulates overseas. After visiting several overseas posts, the Panel

concluded that the condition of U.S. posts and missions abroad was

unacceptable and recommended eight major types of changes, which,

according to its report, would improve the efficiency and effectiveness of

our overseas presence.4

 Specifically, the panel directed the President, the

Secretary of State, and congressional leaders to form a partnership to

implement the following eight recommendations:

• improve security and establish accountability for security at our overseas

facilities;

 4

America’s Overseas Presence in the 21st Century: The Report of the Overseas Presence

Advisory Panel, U.S. Department of State (November 1999). Three other studies of the

condition of our overseas posts have been performed during the last 3 years. These studies

are Road Map for National Security: Imperative for Change, The United States

Commission on National Security/21st Century (February 15, 2001); Independent Task

Force Report: State Department Reform, Council on Foreign Relations (February 2001);

and Equipped for the Future: Managing U.S. Foreign Affairs in the 21st Century, The

Henry L. Stimson Center (October 1998). All these studies cited the need to upgrade

information technology at U.S. overseas locations.

Background
Page 5 GAO-02-41 State-Led Modernization Initiative

• create the right size and shape for our overseas presence, ensuring that

posts are not over- or under-staffed;

• ensure that U.S. facilities are quickly and efficiently renovated and are

properly maintained and managed on an ongoing basis;

• adopt best practices for human resources management, such as improving

recruitment and expanding training and promotion opportunities;

• invest in up-to-date information technology, including providing Internet

access and e-mail to all staff;

• improve and reinforce consular services, such as same-day processing of

visas;

• enact cost- and time-saving administrative reforms, such as consolidating

some functions (e.g., accounting practices) and automating others; and

• reinforce the responsibilities and authorities of Ambassadors, who

represent the interests of the President of the United States.

Because State is the lead agency for foreign affairs,5

 President Clinton

directed the former Secretary of State, on February 10, 2000, to lead a

cabinet-level committee to implement the Panel’s recommendations. As

part of the former Secretary’s response, the Under Secretary for

Management formed an interagency Overseas Presence Committee (the

Committee) in February 2000 to address three of the eight

recommendations. The Committee then established three subcommittees

to implement the Panel’s recommendations: Overseas Facilities,

Interagency Rightsizing, and Interagency Technology.6

 (See fig. 1 for the

Committee’s structure and table 1 for allocation of responsibilities.)

 5

As the lead agency representing U.S. interests overseas, State is vested with a wide range

of responsibilities, including formulating U.S. policy on diverse international issues,

coordinating and implementing U.S. government programs and activities overseas, and

influencing other countries to adopt policies and practices consistent with U.S. interests.

6

According to State’s Director of the Office of Management, Policy, and Planning, this

organizational arrangement is currently under review for potential restructuring at a later

date.
Page 6 GAO-02-41 State-Led Modernization Initiative

Figure 1: Management Structure Established to Address Several Panel Recommendations

a

FASI = Foreign Affairs Systems Integration.

Source: State.
Page 7 GAO-02-41 State-Led Modernization Initiative

Table 1: Summary of Management Responsibilities and Functions of the Overseas Presence Committee and its Entities

Entity Responsibility/Function

Overseas Presence Committee Established to implement the Panel’s recommendations (in its 1999 report) regarding

overseas facilities, interagency rightsizing, and information technology. Chaired by State’s

Under Secretary for Management.

Overseas Facilities Subcommittee Tasked with seeking new ways to manage and finance overseas facilities, such as

evaluating the option of charging rent to the occupants of overseas diplomatic facilities.

Chaired by the Deputy Assistant to the Secretary of State.

Interagency Rightsizing Subcommittee Directed to address issues such as the size and shape of overseas posts, the streamlining

of every mission, the reallocation of all personnel, and budget savings from reducing the

size of overstaffed posts. Chaired by the U.S. Deputy Representative to the United

Nations.

Interagency Technology Subcommittee

(ITS)a

Charged with defining the high-level functional requirements, selecting the specific

enabling strategies, and identifying the needed funding for the common overseas system

through its two permanent working groups—knowledge management and information

technology. Chaired by State’s Chief Information Officer (CIO) and includes the CIOs of

State’s agency partners for this initiative.

Knowledge Management Working GroupbEstablished within the ITS to facilitate information sharing to support a collaborative work

environment. Charged with defining the operational requirements for knowledge

management, recommending enabling strategies, identifying resources to implement those

strategies, providing oversight and guidance, and identifying needed funding. Led by

State’s Deputy CIO, who is also State’s Chief Knowledge Officer, and includes

representatives from State and its agency partners.

Information Technology Working GroupbEstablished within the ITS to provide the foreign affairs community with a common,

interoperable technology and infrastructure. Intended to identify commercial- and

government-off-the-shelf applications and Internet and Internet-like technologies to support

interagency collaboration. This effort is to include analyzing requirements for a common

overseas platform as well as related planning tasks such as design, architecture, and

acquisition. Led by State’s Deputy CIO/Chief Knowledge Officer and comprises

representatives from State and its agency partners.

Interagency Capital Financing Working

Groupc

Established within the ITS to develop a capital financing strategy for funding the

implementation of the Panel’s IT recommendations. Led by State’s Deputy CIO/Chief

Knowledge Officer and comprises representatives from State and six of its eight agency

partners.

FASI Architectural Working Group Established within the ITS to develop and implement the enterprise architecture for the

common overseas system, ensuring compliance with State and its agency partners’

existing enterprise architectures. Led by State’s Director for Standards, Policy, Planning,

and Architecture.

FASI Program Office Created to plan and implement the common overseas platform. Using the user and system

requirements devised and prioritized by the Knowledge Management and Information

Technology working groups, this team is to design and deploy an unclassified,

interoperable information technology infrastructure and knowledge management system

that is intended to address the recommendations in the Panel’s report. Led by State’s

Deputy CIO/Chief Knowledge Officer.

a

The efforts of this subcommittee are the primary focus of this report.

b

These working groups were combined in April 2001.

c

According to State, membership for this working group currently consists of representatives from the

U.S. Agency for International Development, the Peace Corps, and the Departments of Defense,

Treasury, Justice, and Agriculture.
Page 8 GAO-02-41 State-Led Modernization Initiative

In October 2000, the then Secretary of State reported on the progress

made in addressing the Panel’s recommendations for seven of the eight

recommendations.7 In response to the recommendation to invest in up-todate information technology, State reported that it had

• established an acquisition and deployment strategy, preliminary system

requirements, a high-level implementation plan, and an architectural

concept for a common overseas knowledge management system

infrastructure;

• established September 2000 as the goal for completing a formal program

management plan, with a goal of revising the plan by April 2001 after

completing evaluation of a prototype system; and

• established December 2001 as the goal for completing system pilot testing

at two posts.

On August 8, 2001, State issued its final status report on its efforts to

address the Panel’s recommended changes to the Chairman of the Senate

Committee on Foreign Relations. With regard to the recommendation to

upgrade information technology, State reported that the interagency group

has developed and is testing a prototype of the common overseas system

and that a field test of the system’s infrastructure is being conducted at

pilot posts in Mexico and India. In addition, State officials told us that the

date for completing system prototyping has been delayed by 13 months to

May 2002, and the date for completing pilot testing at two posts has been

delayed by 9 months to September 2002. Also, the three vendors selected

to develop system prototypes are not scheduled to deliver these systems

to State for evaluation until November 2001.

The modernization program is intended to put in place a common overseas

knowledge management system to facilitate unclassified

information/knowledge sharing among foreign affairs agencies. Key

system capabilities are to include, among other things, interagency e-mail,

post-specific news and information web-based links, crisis coordination

support, policy formulation support, and various administrative functions

(e.g., accounting, contract management, training, and travel). According to

State’s draft program management plan, the department will acquire and

deploy system functionality in each of these areas in three increments, or

three prioritized groupings of user requirements, which State calls pilot,

 7

The report did not address the Panel’s recommendations for enacting cost- and timesaving administrative reforms.

Common Overseas System

Modernization: A Brief

Description
Page 9 GAO-02-41 State-Led Modernization Initiative

full-scale implementation, and deferred. (See table 2 for planned system

functionality by increment.)

Table 2: Planned System Functionality by Increment

Increment 1: Pilot

Type: General user requirements

--accommodate Sensitive But Unclassified and Unclassified information and

 processing

--ensure access to system during working hours at all locations

--support mobile/traveling users

--provide a powerful “smart” search capability

--support the ability to identify experts in foreign policy, technical, and

 administrative disciplines (i.e., a knowledge management locator service)

--enable “self service” processing (e.g., the ability for users to update records and

 initiate transactions)

Type: Interagency e-mail

--provide antivirus processing

--enable delivery of e-mail to the user’s existing agency e-mail system

--provide access to e-mail addresses and other basic information on participants in

 the Overseas Presence Advisory Panel system

--enable end users to create customized directories and group addresses

--provide e-mail capability between post and non-U.S. government entities

--support private and public mailing lists and list management services

Type: News and information: provide access to

--administrative information (e.g., financial bulletins, per-diem rates, travel

 regulations)

--post or regional newsletters and bulletin boards

--“news gathering and analysis” activities

--news “clipping” services

--issue-oriented databases

Type: Crisis coordination: provide support for

--identifying and locating employees at a given post

--collaborating and conferencing to support work groups and information sharing

--coordinating activities (e.g., transportation, medical treatment)

Type: Policy formulation: provide support for

--researching issues related to a given policy

--publishing of current policies and decisions

--collaboratively developing policy documents, guidelines, processes, and

 decisions

--identifying responsible departments and persons for a given policy area

Type: Administration: provide support for

--international Cooperative Administrative Support Services System

--contacting management through a world-wide, on-line staff directory

--coordinating and managing calendars and schedules
Page 10 GAO-02-41 State-Led Modernization Initiative

Increment 2: Full-scale implementation

Type: General user requirements

--provide support for teams with non-U.S. government members

Type: Interagency e-mail: provide

--e-mail forwarding (inside or outside the collaboration zone)

--integration with workflow tool(s)

--embedded URLs (Internet and Intranet)

Type: News and information: provide access to

--local/regional translation services

--specialized subscription services

--commercial telephone directories

--geographic databases

--mission performance plan data

--telegrams (current and archival)

Type: Crisis coordination

--provide support for notifying and disseminating information, orders, and

 instructions to government and U.S. citizens

Type: Policy formulation

--provide support for bureau and mission performance plans

Type: Administration: provide support for

--training and human resources

--procurement

--travel planning and management

--budgeting and financial management

Increment 3: Deferred

Type: General user requirements

--support access from electronic organizers, cellular telephones, personal digital

 assistants, etc.

--provide access to agency (noncommon-overseas) systems

Type: Administration: provide support for

--motor-pool scheduling

--inventory and equipment maintenance

--property management

--applications development

Source: State.

The system is to use commercially available, Internet-based technology to

enable users to access the system from any location using their respective

agencies’ existing networks and operating platforms. The desktop Internet

browser is intended to be the connection to the common overseas system

and its knowledge management tools. A common website is to serve as the

portal to the customized database applications, files, and communication

tools to be shared across the foreign affairs community’s diverse,

distributed environment. (See fig. 2 for a simplified diagram of the

system.)
Page 11 GAO-02-41 State-Led Modernization Initiative

Figure 2: Simplified Diagram of Target Overseas Knowledge Management System

Source: State.
Page 12 GAO-02-41 State-Led Modernization Initiative

In 1999, the Panel estimated the cost to acquire and deploy this

unclassified system at about $200 million, assuming the use of

commercially available technology. About a year later, State estimated that

the investment in the system through fiscal year 2004 would be about $271

million. State plans to update this cost estimate. According to State’s draft

program management plan, the estimated cost to perform system

prototyping and pilot testing is $18.2 million.

The conference report accompanying the Departments of Commerce,

Justice, and State, the Judiciary, and Related Agencies Appropriations Act,

2001, stated the conference committee’s expectation that State would

spend $17 million to carry the program through pilot testing.8

 State has met

with subcommittee staff to keep them informed of its plans and progress,

and State has agreed to submit a spending plan for the pilot project, as

directed by the conference report. Prototyping activities are currently

scheduled for completion in May 2002.

In June 2000, we testified that State was in the early stages of planning for

the overseas-presence system modernization—establishing preliminary

milestones, developing rough cost estimates, and formulating a program

management plan.9

 At that time, we expressed concern regarding (1)

State’s ability to obtain agreement internally as well as from the foreign

affairs partner agencies in devising and deploying a common technology

solution and (2) State’s lack of an approved architecture to guide and

constrain its own IT investments. In our testimony, we stated, among other

things, that it was important for State to (1) carefully plan the initiative

establishing realistic goals and milestones, (2) install the needed

management and oversight accountability to properly guide the acquisition

of the system, and (3) anticipate the steps needed to gain the full

cooperation of its agency partners.

 8

H.R. Conf. Rep. No. 106-1005, at 294 (2000) states, in part, that $17 million shall be for a

pilot project to establish a common technology platform at overseas posts pursuant to the

recommendations of the Overseas Presence Advisory Panel. The conference report also

directs that State submit a spending plan for this pilot project.

9

Foreign Affairs: Effort to Upgrade Information Technology Overseas Faces Formidable

Challenges (GAO/T-AIMD/NSIAD-00-214, June 22, 2000).

Recent Review of Common

Overseas System

Modernization Program

Noted Management

Challenges
Page 13 GAO-02-41 State-Led Modernization Initiative

Office of Management and Budget (OMB) guidance outlines risk reduction

principles that are consistent with best commercial practices and, if

effectively implemented, can increase the probability that system

modernization programs will succeed.10 Among other things, this guidance

supports (1) prototyping system components before acquiring or

developing the operational system, (2) using commercially available

technology, and (3) securing system stakeholders’ involvement and

agreement on products such as management plans and system

requirements.

State’s near-term plans generally provide for each of these areas of

guidance. In particular, State is currently focused on acquiring and

evaluating system prototypes that are to employ either commercially

available and/or government-used hardware and software products. On

July 5, 2001, State selected three vendors, each of which was to develop a

prototype system by October 2001. However, according to a State official,

because of security and infrastructure concerns identified during testing

that the vendors must address, the systems are now scheduled for delivery

in November 2001. In addition, State planned to evaluate the delivered

prototype systems using evaluation criteria that were to be developed by a

private firm by September 28, 2001.11 While an initial draft of the criteria

was provided on that date, according to a State official, revisions were

needed, and the deadline for finalizing the evaluation criteria was changed

to mid-November 2001. Although these events will delay the deadline for

completing the prototype evaluation one month to May 2002, piloting

activities will not be affected and will begin as scheduled. Once the

prototype system solutions have been evaluated, State and its agency

partners plan to use the best capabilities from each prototype to refine

user requirements and develop a specification for the foundational system

capabilities that will be pilot tested. State plans to pilot test these

foundational capabilities in India and Mexico between April and

September 2002. According to State, no commitments have been made to

proceed beyond this pilot test, and any decisions about how best to

proceed will have to involve agency partners, OMB, and relevant

congressional committees.

 10Office of Management and Budget (OMB) Circular A-130, Management of Federal

Information Resources (November 30, 2000) and Capital Programming Guide, Version

1.0, OMB Circular A-11, part 3 (July 2000).

11State officials told us that the deadline for the criteria was changed from September 28,

2001, to mid-November 2001 because of the events of September 11, 2001.

State’s Near-Term

Plans Are Focused on

Risk Reduction

Activities
Page 14 GAO-02-41 State-Led Modernization Initiative

In addition to prototyping the system and using commercially available

products, State has also taken steps to engage other foreign affairs

agencies in its prototyping activities, including securing stakeholder

involvement and buy-in. Specifically, responses from the seven

modernization program partner agencies that responded to our survey

showed that they were generally satisfied with State’s efforts to engage

them in near-term program activities.12 Additionally, four of these agencies

stated that State had informally obtained their concurrence on such items

as program plans and system requirements. As for the other three

agencies, one stated that while it had reviewed some of the program

documentation, it had not participated in an approval process; another

stated that it had reviewed program documentation for informational

purposes only and was unaware of any approval process; and the third

stated that it had not reviewed any of the program documentation.

Nevertheless, we found that these and other interagency meetings, such as

program working group meetings,13 have occurred and provided forums

for informally engaging agency partners. In general, agency partners

attended these meetings and participated in the program. For example,

two agency partners have assigned staff to the FASI Program Office.14

Available documentation and State CIO officials’ statements confirm that

agency partners’ buy-in to program decisions, such as agreement on the

requirements for the prototype and approval of plans, is being

accomplished informally. According to State, representatives for the

partner agencies agreed that their lack of opposition or comments equated

to concurrence on actions, plans, and/or documents, thereby eliminating

the need to obtain actual signatures to indicate approval. Although State

could not provide us with any written agreements to this effect, revised

minutes of an Interagency Technology Subcommittee meeting reflected

this informal agreement.

 12As of November 9, 2001, the U.S. Agency for International Development, the Peace Corps,

and the Departments of Agriculture, Justice, Transportation, Commerce, and Treasury had

responded to our survey; the Department of Defense had not.

13These meetings include the Knowledge Management Working Group, the Information

Technology Working Group, the Architecture Working Group, and the Interagency Capital

Financing Working Group.

14Treasury has assigned an employee to State’s Knowledge Management Team, and the

Justice Department has detailed an individual to assist State in developing the overseas

presence enterprise architecture.
Page 15 GAO-02-41 State-Led Modernization Initiative

The Clinger-Cohen Act of 1996, in combination with OMB guidance

implementing the act, provide federal agencies with a framework for

effectively managing modernization efforts that are consistent with best

commercial practices.15 Among other things, OMB requires (1) a clear

definition of program management roles, including an explicit definition of

responsibilities and accountabilities for such things as funding, investment

decisionmaking, requirements management, architecture definition and

integration with legacy systems, and testing; (2) use of incremental

investment management principles to minimize the enormous risk

inherent in large modernization programs that involve many things to be

done over many years; (3) the development, maintenance, and

implementation of an enterprise architecture, which is an institutional

blueprint for guiding operational and technological modernization; and (4)

the use of effective IT project management processes, such as processes

for managing software-intensive system acquisition projects.

For its overseas knowledge management system modernization program,

State program officials told us that they plan to establish and implement

each of these management controls. However, much remains to be

accomplished because thus far State has devoted its attention to system

prototyping activities, and, according to State officials, they first had to

brief congressional staff earlier in fiscal year 2001 on the use of

appropriated funds before establishing these management controls.16

It is essential that these controls be effectively implemented on the

overseas-presence system modernization program before State proceeds

with its acquisition of the operational system. To do less would

unnecessarily increase the risk that needed system capabilities will not be

delivered on time and within budget.

 15Clinger-Cohen Act of 1996, Public Law 104-106 and OMB Circular A-130, Management of

Federal Information Resources (November 30, 2000).

16Consistent with congressional direction to submit a spending plan for the pilot project,

State briefed congressional staff on its plans for the pilot project and its proposed use of

the funds before it began spending the $17 million.

Much Remains to be

Done Before

Acquiring the

Operational System
Page 16 GAO-02-41 State-Led Modernization Initiative

OMB guidance recognizes the importance of fully and explicitly defining

and ensuring understanding of program stakeholders’ areas of

responsibility and accountability in managing and funding a program.

Similarly, the Software Engineering Institute (SEI), recognized for its

expertise in managing software-intensive systems, emphasizes the

importance of clearly designating roles and responsibilities on system

acquisition programs. While doing so is essential on any program, it is

particularly critical for programs like the overseas-presence knowledge

management modernization program because its success depends on the

effective interplay of multiple federal agencies. Without well-defined,

understood, and agreed-to roles in such areas as funding and acquisition

management (e.g., investment decisionmaking, requirements management,

architecture definition and integration with legacy systems, and testing),

State cannot, for example, be assured that necessary resources will be

provided (e.g., financial, human capital, and technological) and critical

tasks executed.

According to State, the roles, responsibilities, and accountabilities of

foreign affairs agencies have been formally defined. However, State did

not provide us with documentation to support this position, as we

requested. Furthermore, agency partners that responded to our survey

stated that their respective roles, responsibilities, and accountabilities for

this initiative have not yet been formally defined.

According to State, as the principal foreign affairs agency, its role is to

provide administrative and management leadership and support for nearterm prototype and pilot-testing activities, and it is the agent for bringing

the foreign affairs agencies together. However, State said that it is not

responsible for engaging agency partners as part of the near-term

prototype and pilot-testing activities.

As noted earlier, given the stated purpose and scope of these near-term

activities and State’s leadership role, this definition of roles is not

currently a significant issue. However, it is essential that State and its

agency partners fully define agreed-to roles, responsibilities, and

accountabilities for program funding and management before moving

beyond planned near-term activities. To do less would unnecessarily put

the modernization program at risk.

Clear Definition of

Stakeholders’

Accountability Is Critical

to Proceeding Beyond

Pilot Testing
Page 17 GAO-02-41 State-Led Modernization Initiative

Incremental investment management involves three fundamental

components: (1) developing/acquiring a large system in a series of smaller

projects or system increments; (2) individually justifying investment in

each increment on the basis of costs, benefits, and risks; and (3)

monitoring actual benefits achieved and costs incurred on ongoing

increments and modifying subsequent increments/investments to reflect

lessons learned. Using this approach, agencies can avoid discovering too

late that their systems are not cost beneficial and reduce the risks

associated with large, expensive projects. This approach allows overlap

and smooth transition among increments because lessons learned from

actual results of ongoing increments would be monitored and evaluated

continuously so that these results will be available for use in defining and

justifying future increments.

The Clinger-Cohen Act of 1996 requires agencies to follow, to the

maximum extent practical, an incremental approach to investing in IT

projects. Also, OMB policy requires that investments in major systems be

implemented in increments, with each increment delivering measurable

benefits.17 More specifically, OMB’s Capital Programming Guide18 describes

the use of modular contracting, or incremental investment, including its

application and benefits. In particular, OMB states that project increments

should provide for the following:

• Separability: Each increment should be fully funded, have substantial

programmatic use that is not dependent on any subsequent increment, and

be capable of performing its principal functions, even if no subsequent

increments are acquired.

• Interoperability: Each increment should comply with a common

architecture or commercially acceptable technology standards and should

be compatible with and capable of being integrated with other increments.

• Performance requirements: Each increment’s performance requirement

should be consistent with the performance requirements of the completed

overall system and should address interface requirements with other

increments.

 17OMB Circular A-130 (November 30, 2000); Capital Programming Guide, Version 1.0,

OMB Circular A-11, part 3 (July 2000), pp. 545-572.

18Capital Programming Guide, Version 1.0, OMB Circular A-11, part 3, Supplement,

Planning, Budgeting, and Acquisition of Capital Assets (July 1997), pp. 35-37.

Incremental Investment

Management Could

Reduce Overall Program

Risks
Page 18 GAO-02-41 State-Led Modernization Initiative

In short, incremental investment helps to mitigate the risks inherent in

acquiring/developing large systems by dividing a single, large program into

smaller, independently useful components with known and defined

relationships and dependencies. It is well understood that making

investment decisions up front on large-scale, long-term programs is risky

because their economic justification is based on costs, benefits, and risks

that are difficult to forecast reliably, partially because later increments are

not well understood or defined and partially because they are subject to

change in light of experience on nearer term increments and changing

business needs. Through incremental investment management,

organizations can

• reduce the level of program risk and complexity faced at any one time by

spreading this risk and complexity across a series of smaller investments;

• permit the delivery of some part of the expected business value earlier

rather than waiting until later for the total, but more uncertain, business

value;

• continuously monitor and evaluate the delivery of cost and benefit

expectations on ongoing increments and use this information to better

define and economically justify these increments; and

• permit later increments to exploit technology advances or accommodate

evolving business needs.

According to modernization officials, preliminary plans provide for

acquiring and investing in the overseas knowledge management system in

three increments: pilot, full-scale implementation, and deferred. However,

State has not yet established specific plans and management processes for

acquiring the operational system because, according to State’s CIO, it does

not have congressional authorization to do so. We were unable to

corroborate State’s position as to why it has not established these plans

and processes.

In addition, modernization officials stated that the ongoing system

prototyping and planned pilot-testing activities were not economically

justified as a separate increment. However, these officials noted that all

future incremental investments in the system will be economically

justified, and the results and lessons learned from prior increments will be

used to define and make investment decisions about future increment(s).

Additionally, they stated that future increments will also be evaluated to

ensure that return-on-investment commitments are met.

It is critical for State to follow through on its stated commitments to

implement incremental investment practices. Otherwise, it risks making a
Page 19 GAO-02-41 State-Led Modernization Initiative

single, monolithic investment decision for the entire program. This “all or

nothing” approach to investing in IT has historically resulted in agencies

investing huge sums of money in systems that do not provide

commensurate benefits, and thus has been abandoned by successful

organizations. The need to avoid this pitfall was a major impetus for the

investment management reforms of the Clinger-Cohen Act.

Enterprise architectures are essential tools for effectively and efficiently

reengineering business processes and for implementing and evolving their

supporting systems. Enterprise architectures systematically capture—in

useful models, diagrams, and narrative—the full breadth and depth of the

mission-based mode of operations for a given enterprise, which can be (1)

a single organization or (2) a functional or mission area that transcends

more than one organizational boundary (e.g., financial management,

acquisition management, or overseas-presence foreign affairs activities).

An architecture describes the enterprise’s operations in both (1) logical

terms, such as interrelated functions, information needs and flows, work

locations, and system applications, and (2) technical terms, such as

hardware, software, data, communications, security attributes, and

performance standards. It provides these perspectives for both the

enterprise’s current, or “as is,” environment and its target, or “to be,”

environment; it also provides an IT capital investment road map for

moving between the two environments.

The development, implementation, and maintenance of enterprise

architectures are recognized hallmarks of successful public and private

sector organizations. Managed properly, an enterprise architecture can

clarify and help optimize the interdependencies and interrelationships

among an organization’s business operations and the underlying IT

infrastructure and applications that support these operations. Our

experience with federal agencies has shown that attempting to define and

build major IT systems without first completing an enterprise architecture

often results in systems that are duplicative, not well integrated,

unnecessarily costly to maintain and interface, and not effectively

optimizing mission performance.

Federal CIO Council guidance defines a set of recognized key practices

(management structures and processes) for developing and implementing

Enterprise Architectures

Are Essential Tools for

Guiding and Constraining

Investment Decisions
Page 20 GAO-02-41 State-Led Modernization Initiative

an enterprise architecture that are employed by successful public and

private sector organizations.19 Among other things, these practices include

the following:

• Because the architecture is a corporate asset for systematically managing

institutional change, the head or leader of the enterprise should support

and sponsor the architecture, giving it a clear mandate in the form of an

enterprise policy statement. Such support is crucial to gaining the

commitment of all organizational components of the enterprise, all of

whom should participate in developing and implementing the enterprise

architecture.

• The enterprise architecture effort should be directed and overseen by an

executive body, empowered by the head(s) of the organization(s), with

members who represent all stakeholder organizations and have the

authority to commit resources and to make and enforce decisions for their

respective organizations.

• The enterprise architecture effort should be led by a Chief Architect who

reports to the enterprise CIO, and it should be managed as a formal

program. A formal program entails creating a program office; committing

core staff; implementing a program management plan that details work

breakdown structure and schedules; allocating resources and tools;

performing basic program management functions (e.g., risk management,

change control, quality assurance, configuration management); and

tracking and reporting progress against measurable goals.

• The enterprise architecture should conform to a specified framework.

Although State modernization officials have acknowledged the importance

of having an overseas-presence enterprise architecture and using this

architecture to guide and constrain investment in a common overseas

knowledge management system, to date substantive progress in this area

has not occurred. Currently, State and its agency partners have yet to

either develop the architecture or establish effective management

structures and processes for developing, maintaining, and implementing

one, such as those specified in federal CIO Council guidance. More

specifically, although State established an Enterprise Architecture Study

Group in December 2000, the group met only twice and progressed no

further than drafting a charter and setting a goal of completing the

architecture by November 2001.

 19Chief Information Officer Council, A Practical Guide to Federal Enterprise Architecture,

Version 1.0 (February 2001).
Page 21 GAO-02-41 State-Led Modernization Initiative

On June 8, 2001, State designated a Chief Architect to lead this effort. In

July 2001, State reconvened the group, renaming it the Foreign Affairs

Systems Integration Architectural Working Group, and set February 15,

2002, as the new goal for completing the architecture. Since reconvening,

the working group has held three meetings to begin planning for

development of the overseas-presence enterprise architecture and

developed a draft charter. The working group has also begun surveying its

agency partners to obtain general information on architectural

requirements of overseas locations. State’s ongoing prototyping and

planned piloting efforts, which are intended to provide information to

better understand business and technology requirements and alternate

approaches, should facilitate development of the architecture.

It is essential that State and its partner agencies effectively develop,

maintain, and implement an enterprise architecture for its overseas

modernization program. The federal CIO Council architecture

management guide provides a road map for doing so. Our experience with

system modernization programs in other federal agencies has shown that

attempting to define and build major systems without first completing an

enterprise systems architecture often results in systems that are

duplicative, not well integrated, unnecessarily costly to maintain and

interface, and not effectively optimizing mission performance.20

The Clinger-Cohen Act requires agency CIOs to establish effective IT

management processes, such as processes for acquiring software.21 The

SEI, recognized for its expertise in software process controls, has

developed a model approach for software acquisition. This model outlines

the key acquisition process management controls that, if implemented

effectively, can greatly increase the chances of acquiring softwareintensive systems that provide promised capabilities on time and within

budget. The key processes are as follows:

 20See, for example, Air Traffic Control: Complete and Enforced Architecture Needed for

FAA Systems Modernization (GAO/AIMD-97-30, February 3, 1997) and Customs Service

Modernization: Architecture Must Be Complete and Enforced to Effectively Build and

Maintain Systems (GAO/AIMD-98-70, May 5, 1998).

21Clinger-Cohen Act of 1996, Public Law 104-106.

Disciplined Acquisition

Management Processes

Can Further Reduce

Program Risks
Page 22 GAO-02-41 State-Led Modernization Initiative

• Software Acquisition Planning: Ensures that reasonable planning for the

software acquisition is conducted and that all aspects of the total software

acquisition effort are included in these plans.

• Solicitation: Ensures that a request for proposal that delineates a project’s

software-related requirements is prepared, and, consistent with relevant

solicitation laws and regulations, that a contractor that can most costeffectively satisfy these requirements is selected.

• Requirements Development and Management: Establishes and maintains

a common and unambiguous definition of software requirements among

the acquisition team, the system users, and the software development

contractor.

• Project Office Management: Provides for management of the activities

within the project office and supporting contractors to ensure a timely,

efficient, and effective software acquisition.

• Contract Tracking and Oversight: Ensures that the software development

contractor performs according to the terms of the contract; needed

contract changes are identified, negotiated, and incorporated into the

contract; and contractor performance issues are identified early, when

they are easier to address.

• Evaluation: Determines whether the acquired software products and

services satisfy contract requirements prior to acceptance.

• Transition and Support: Provides for the effective and efficient “hand-off”

of the acquired software products to the support organization responsible

for software maintenance.

• Acquisition Risk Management: Identifies risks as early as possible and

adjusts the acquisition to mitigate those risks.

Within these key processes, SEI identifies key practices that are needed to

effectively execute each key process. Among others, these key practices

include: (1) designating responsibility for activities and assigning adequate

resources; (2) having a written policy; (3) developing, documenting, and

adhering to a plan; (4) performing management review activities; and (5)

measuring the status of key activities, and using these measurements to

make decisions.

According to State program officials, the modernization program will be

managed in accordance with State’s methodology, Managing State Projects

(MSP). MSP specifies major project management activities to be

performed, the products to be generated, and the control gates (i.e.,

decision points) to be used to ensure that projects are ready for the next
Page 23 GAO-02-41 State-Led Modernization Initiative

phase in their life cycle. We analyzed State’s MSP and found that it is

consistent with best practice models, such as SEI’s acquisition model.22

For its near-term prototyping and pilot-testing activities, which do not

demand the level of management process control rigor and discipline

associated with acquiring a system solution intended for operational use,

State is employing adequate acquisition management process controls. For

example, State has defined a work breakdown structure and schedule to

guide near-term activities. Further, State CIO officials told us that

management plans relating to key process areas have been drafted,

although they did not provide us with copies of the draft plans, as we

requested. According to State, these plans will be available when finalized

and approved. These officials also told us that program management is

using two automated tools to track the status of program activities. We

verified that State has been using one of these tools to track programrelated tasks.23

According to State, it has not yet established acquisition management

controls for the future operational version of the overseas-presence

knowledge management system because it does not have congressional

authorization to plan beyond the pilot project for this initiative. Beyond

intentions to employ those controls already in place for key processes for

the prototype and pilot efforts (i.e., designating a responsible official and

having a written policy), State has important practices that will need to be

established before it is ready to acquire the operational system. Table 3

summarizes the key process areas for managing acquisition of the

operational version of the system that already exist and those that will

need to be established.

 22Based on a comparison of the MSP’s major process areas with SEI’s key processes.

23We did not pursue verification of State’s use of the other tool.
Page 24 GAO-02-41 State-Led Modernization Initiative

Table 3: Summary of Needed Process Controls for Managing the Acquisition of the Operational Version of the System

Process area

Designate

responsible

official

Assign

adequate

resources

Have a

written

policya

Document a

plan outlining

activities

Perform

management

review of

activities

Measure and

report on

status of

activities

Software acquisition

planning

√ ÿ √ ÿÿÿ

Solicitation √ ÿ √ ÿÿÿ

Requirements

development and

management

√ ÿ √ ÿÿÿ

Project office

management

√ ÿ √ ÿÿÿ

Contract tracking and

oversight

√ ÿ √ ÿÿÿ

Evaluation √ ÿ √ ÿÿÿ

Transition and support √ ÿ √ ÿÿÿ

Acquisition risk

management

√ ÿ √ ÿÿÿ

√—Satisfied

ÿ—Not satisfied

a

Based on State’s plans to adhere to its Foreign Affairs Manual, which requires implementation of its

MSP methodology.

Source: State.

It is imperative that State ensures that it fully implements key acquisition

management process controls, as defined in its MSP and the SEI model, on

its overseas knowledge-management modernization program before

acquiring the operational system. Our reviews of other agency

modernization efforts have shown that failure to implement rigorous and

disciplined acquisition processes on a given program or project can lead to

systems that do not perform as intended, cost more than expected, and are

not delivered on time.24

 24See, for example, Customs Service Modernization: Serious Management and Technical

Weaknesses Must Be Corrected (GAO/AIMD-99-41, February 26, 1999) and Land

Management Systems: BLM’s Actions to Improve Information Technology Management

(GAO-01-282, February 27, 2001).
Page 25 GAO-02-41 State-Led Modernization Initiative

The scope of the near-term activities for State’s overseas-presence

knowledge management modernization program, as described by State

CIO officials and described in draft plans, is appropriately limited to

system prototyping and associated pilot testing of a version of the

operational system. In light of this scope, as well as State’s leadership role

in managing and funding these near-term efforts and its efforts to date to

bring its agency partners together, there is currently an adequate level of

(1) explicit definition and understanding surrounding the overseaspresence agencies’ respective responsibilities and accountabilities, (2)

agency involvement in the program, and (3) State’s management control

discipline.

However, before State can effectively acquire and deploy the operational

knowledge management system, it needs to more explicitly define

stakeholder agencies’ program management and funding roles, as well as

establish more rigorous and disciplined management controls.

Accordingly, it is critical for State to also move aggressively in the near

term to (1) clearly define, through written agreements, the respective

management and funding responsibilities and accountabilities of the

program’s agency partners and (2) institute key modernization

management controls, including incremental investment management

practices, enterprise architecture governance, and software acquisition

management rigor and discipline. While State has verbally committed to

addressing these areas, much remains to be done before State and its

agency partners will be ready to proceed beyond prototyping and pilot

testing and begin acquiring the system solution intended for operational

deployment and use. Attempting to acquire the operational system

solution without these controls risks not delivering needed system

capabilities on time and within budget.

To provide for an explicit definition and understanding of the respective

roles of all overseas-presence modernization agency stakeholders, we

recommend that the Secretary of State develop and submit a proposal for

OMB and/or congressional action, whichever the Secretary deems

appropriate, within 60 days of the date of this report that clearly defines

and assigns responsibilities and accountabilities for each overseaspresence agency regarding the knowledge management system

modernization program. We further recommend that the Secretary direct

the Under Secretary for Management to ensure that this proposal (1) sets

expectations for each overseas-presence agency, including the State

Department, relative to modernization management and funding and (2) is

Conclusions

Recommendations for

Executive Action
Page 26 GAO-02-41 State-Led Modernization Initiative

coordinated with and agreed to by the Under Secretary’s counterparts in

each of the partner agencies.

To ensure that key system modernization management controls are

implemented before the operational system is acquired, we recommend

that the Under Secretary of State for Management direct State’s CIO, as

chair of the Interagency Technology Subcommittee, to ensure that the

subcommittee allows only architecturally compliant knowledge

management system investments to be approved and funded, unless the

subcommittee issues a written waiver in response to a written

justification. Furthermore, we recommend that the Under Secretary direct

the CIO to ensure that architecturally compliant investments in the system

be made incrementally. To this end, the CIO should ensure that processes

and structures are established for (1) acquiring the system in a series of

smaller system increments; (2) individually justifying investment in each

increment on the basis of costs, benefits, and risks; and (3) monitoring

actual benefits achieved and costs incurred on ongoing increments and

modifying subsequent increments to reflect lessons learned.

To further ensure that key management controls are implemented on the

overseas-presence knowledge management system modernization, we

recommend that the Secretary of State designate as a modernization

program priority the development, implementation, and maintenance of an

overseas foreign-affairs presence enterprise architecture. To this end, we

recommend that the Secretary direct the Under Secretary for Management

to establish an enterprise architecture steering committee, chaired by the

Under Secretary and composed of the Under Secretary’s counterparts

from the other overseas-presence agencies. We also recommend that this

steering committee be assigned responsibility and accountability for

ensuring that a complete and approved enterprise architecture is

developed and available to guide and constrain overseas-presence system

modernization efforts. To assist the steering committee, we recommend

that the Under Secretary direct the State CIO to establish an enterprise

architecture project office, headed by the recently named Chief Architect,

to develop and maintain the enterprise architecture and submit

appropriate versions of this architecture to the steering committee for

approval. We further recommend that the State CIO, at a minimum, ensure

that the project office (1) include staff from each of the overseas-presence

agencies; (2) integrate the architecture with these agencies’ respective

operational and systems environments; and (3) adhere, as appropriate,

with the federal CIO Council’s published guidance on managing enterprise

architectures.
Page 27 GAO-02-41 State-Led Modernization Initiative

To further address State’s need to implement modernization management

controls, we recommend that the Under Secretary for Management direct

the State CIO to ensure that the overseas-presence system modernization

office, consistent with State’s project management requirements and

recognized models for effective software acquisition, establishes and

implements rigorous and disciplined acquisition processes, including

processes for acquisition planning, contractor solicitation, requirements

development and management, contractor tracking and oversight, testing

and evaluation, transition to support the acquired system, and risk

management.

Until these recommended modernization management controls are in

place, we also recommend that the Secretary limit further investment in

the overseas-presence knowledge management modernization program to

(1) conducting ongoing prototyping and limited pilot testing and (2)

establishing needed modernization management controls.

In written comments on a draft of this report (reprinted in appendix II),

State’s Acting Chief Financial Officer agreed with our findings and

conclusions that its management controls over near-term prototype and

pilot-testing activities are adequate, but disagreed with our finding and

conclusion that State’s existing controls are not adequate for effectively

managing the more demanding next phase of the program—acquisition

and deployment of the operational system. Furthermore, State disagreed

with our recommendations for instituting stronger management controls

before proceeding to this next phase. According to State, our

recommendations should be deleted from the report because (1) we

reported that management controls over near-term activities are adequate

and thus corrective actions are not needed; (2) except for clearly defining

and agreeing to all agencies’ roles and responsibilities, State’s existing

management controls are adequate for acquiring and deploying the

operational system; and (3) the lead agency for the next phase of the

program is uncertain and thus the recommendations are premature. At the

same time, State said that most of the recommendations might more

reasonably be submitted as suggestions for consideration to the lead

agency for the next phase of the program.

We do not agree with State’s comment that its existing management

controls are adequate for the next phase of the program. As stated in our

report, the nature and purpose of system prototyping and pilot testing are

fundamentally different; thus, the associated risks are less significant than

those related to acquiring a system that is to be operationally deployed. As

Agency Comments

and Our Evaluation
Page 28 GAO-02-41 State-Led Modernization Initiative

a result, it is widely recognized that the level of management control to

effectively put in place an operational system, particularly a system like

the overseas knowledge-management system that involves multiple

agencies, is greater than is needed to prototype or pilot test the system.

Thus, for example, while State’s lack of an enterprise architecture to guide

and constrain near-term prototype and pilot-testing activities does not

introduce significant program risk, it would introduce considerable risk

for State to begin the next program phase without this architecture.

Similarly, while State’s approach to managing its investment in near-term

activities is adequate, effectively acquiring and deploying the operational

system will require rigorous management processes that, for example,

provide for incremental economic justification and measurement of

system return-on-investment, and provide for fully implementing

disciplined acquisition management practices in such areas as managing

system requirements.

We also do not agree with State’s rationale for deleting the report’s

recommendations because these recommendations (1) are not intended to

correct problems with management of the prototype and pilot activities, as

State asserts, but rather to proactively provide a recommended road map

for how to successfully manage the next phase of the program; (2) are

consistent with our conclusions that existing management controls are not

adequate for the next phase of the program; and (3) recognize that prudent

planning requires that agency roles and responsibilities for the next phase

of the program, including designation of the lead agency, be clearly

defined and agreed to during this phase to better ensure program

continuity and to avoid unnecessary delays. Accordingly, the

recommendations are both appropriate and warranted.

State’s written comments, along with our responses, are reproduced in

appendix II.

We are sending copies of this report to the Chairman and Ranking

Minority Members of the Senate Committee on Governmental Affairs;

House Committee on Government Reform; Senate Committee on Foreign

Relations; House Committee on International Relations; the Subcommittee

on Commerce, Justice, State, and the Judiciary, Senate Committee on

Appropriations; and the Subcommittee on Commerce, Justice, State, the

Judiciary and Related Agencies, House Committee on Appropriations. We

are also sending copies to the Director, Office of Management and Budget;

the Secretary of State; the Under Secretary for Management; and the Chief
Page 29 GAO-02-41 State-Led Modernization Initiative

Information Officer, the Department of State. Copies will be made

available to others upon request.

If you have any questions on matters discussed in this report, please call

me at (202) 512-3439 or Cynthia Jackson, Assistant Director, at (202) 512-

5086. We can also be reached by e-mail at hiter@gao.gov and

jacksonc@gao.gov, respectively. Key contributors to this assignment are

listed in appendix III.

Sincerely yours,

Randolph C. Hite

Director, Information Technology Systems Issues
Appendix I: Objectives, Scope, and

Methodology

Page 30 GAO-02-41 State-Led Modernization Initiative

Our objectives were to determine (1) what State’s plans are for a

multiagency, common platform, knowledge management system, and (2)

whether State has effective controls in place for acquiring this system, that

is, defined roles for State and its agency partners and effective controls

over its investment, enterprise architecture, and system acquisition

management processes.

To determine State’s plans for the system, we reviewed State budget

submissions, relevant congressional direction (hearing testimony,

transcript, and related legislation), the results of our prior review,25 the

Overseas Presence Advisory Panel report and recommendations,26 and

available State modernization program management plans and documents.

Additionally, we interviewed State modernization program officials,

including the Deputy Chief Information Officer (CIO)/Chief Knowledge

Officer, Deputy Chief Knowledge Officer, and the Director for Standards,

Policy, Planning, and Architecture, to discuss State’s near- and long-term

plans, as well as representatives of the Overseas Presence Committee and

the Interagency Technology Subcommittee to discuss the status of these

plans and activities, and inter- and intra-agency coordination concerns. To

confirm the extent to which State had engaged its agency partners, we

surveyed State’s eight agency partners to determine whether they (1) were

satisfied with State’s efforts to engage them in the program, (2) had

reviewed and approved the program plans and related documents, and (3)

believed that State was adequately addressing their requirements and/or

information needs. In surveying State’s agency partners, we contacted and

received responses from those individuals designated by their respective

agencies as the authorized representatives for this initiative.

To determine whether State has effective controls in place for acquiring

the system, we did the following.

 25 U.S. Department of State: The Budget in Brief-Fiscal Year 2002; U.S. Department of

State: Congressional Presentation Document—Fiscal Year 2001 Budget Volume I;

Written Testimony of Fernando Burbano, Department of State Chief Information Officer,

presented to The House Committee on International Relations (June 22, 2000); Oversight of

the State Department: Technology Modernization and Computer Security, Hearing before

the Committee on International Relations, House of Representatives, Serial No. 106-171,

June 22, 2000; Office of Management and Budget (OMB) Circular A-130, Management of

Federal Information Resources (November 30, 2000); and Foreign Affairs: Effort to

Upgrade Information Technology Overseas Faces Formidable Challenges

(GAO/T-AIMD/NSIAD-00-214, June 22, 2000).

26America’s Overseas Presence in the 21st Century: The Report of the Overseas Presence

Advisory Panel, U.S. Department of State (November 1999).

Appendix I: Objectives, Scope, and

Methodology
Appendix I: Objectives, Scope, and

Methodology

Page 31 GAO-02-41 State-Led Modernization Initiative

• To determine whether State and its agency partners’ roles, responsibilities,

and accountabilities had been formally defined for each phase of the

program (e.g., prototype, pilot), we reviewed documentation provided by

State (e.g., briefing slides, meeting minutes, working group charters),

interviewed State officials, and surveyed State’s eight agency partners, as

described above.

• To determine State and its agency partners’ plans for managing investment

in the system, we reviewed relevant legislative requirements and

associated OMB guidance,27 as well as applicable State policy and

guidance, on incremental investment management.28 We then obtained and

analyzed documentation outlining State’s investment review process for

selecting, controlling, and evaluating projects since plans are currently for

State to fully fund the project (prototype and pilot). Because

documentation on State’s investment management activities to date and

future plans for this system were not available, we interviewed officials

from the modernization program office [Foreign Affairs Systems

Integration (FASI) Office] and Deputy CIO for Architecture, Planning, and

Regulations Office to determine if State and its partners planned to use an

incremental investment management strategy. We then compared

available information on State’s investment management plans against

relevant federal requirements and guidance to identify if any variances

existed.

• To determine whether an enterprise architecture exists for the system to

guide and constrain investment, we reviewed relevant legislative

requirements and associated OMB guidance,29 as well as federal CIO

Council published guidance30 and applicable State policy and guidance,31

on developing, maintaining, and implementing an enterprise architecture.

 27Clinger-Cohen Act of 1996, Public Law 104-106; OMB Circular A-130 (November 30, 2000);

Capital Programming Guide, Version 1.0, OMB Circular A-11, part 3 (July 1997), pp. 545-

572; and Capital Programming Guide, Version 1.0, OMB Circular A-11, part 3,

Supplement, Planning, Budgeting, and Acquisition of Capital Assets (July 2000), pp. 35-

37.

28U.S. Department of State Foreign Affairs Manual, Section 5 FAM 626.2: Incremental

(February 4, 2000, update).

29Clinger-Cohen Act of 1996, Public Law 104-106 and OMB Circular A-130 (November 30,

2000).

30Chief Information Officer Council, A Practical Guide to Federal Enterprise Architecture,

Version 1.0 (February 2001).

31U.S. Department of State Foreign Affairs Manual, Section 5 FAM 115: IT Management

Oversight Objectives (January 23, 2001, update).
Appendix I: Objectives, Scope, and

Methodology

Page 32 GAO-02-41 State-Led Modernization Initiative

We then reviewed available documentation, including plans and

management structures, a charter, process controls, and architectural

artifacts, as well as interviewing FASI Program Office officials, including

the Director for Standards, Policy, Planning, and Architecture, to

determine the state of any enterprise architecture development and

implementation efforts. We then compared the available information to

relevant federal requirements and guidance to identify if any variances

existed.

• To determine whether State and its agency partners have the needed

capabilities to effectively acquire the system, we obtained and analyzed

documentation on agencywide policies and procedures governing system

acquisition efforts, which State refers to as its Managing State Projects

methodology.32 We then reviewed available best practice models on system

and software acquisition management (e.g., the Software Engineering

Institute (SEI) model33) and compared the primary components of the

Managing State Projects methodology against the primary components of

SEI’s model to determine if any variances existed. SEI’s primary

components include software acquisition planning, solicitation,

requirements development and management, project office management,

contractor tracking and oversight, evaluation, transition and support, and

risk management. Because we found that the primary components of

State’s methodology were consistent with best practices, we obtained and

reviewed available program management and system documentation and

interviewed FASI Program Office officials to determine the extent to

which the methodology was being implemented on the program.

We performed our work at State Department headquarters in Washington,

D.C., from October 2000 through November 2001 in accordance with

generally accepted government auditing standards.

 32Managing State Projects, U.S. Department of State (January 19, 1999).

33Software Acquisition Capability Maturity ModelSM (SA-CMM®). Capability Maturity

ModelSM is a service mark of Carnegie Mellon University, and CMMÆ is registered in the

U.S. Patent and Trademark Office.
Appendix II: Comments From the Department

of State

Page 33 GAO-02-41 State-Led Modernization Initiative

Appendix II: Comments From the

Department of State

Note: GAO comments

supplementing those in

the report text appear at

the end of this appendix.

See comment 1.
Appendix II: Comments From the Department

of State

Page 34 GAO-02-41 State-Led Modernization Initiative

See comment 3.

See comment 2.
Appendix II: Comments From the Department

of State

Page 35 GAO-02-41 State-Led Modernization Initiative
Appendix II: Comments From the Department

of State

Page 36 GAO-02-41 State-Led Modernization Initiative

See comment 8.

See comment 7.

See comment 6.

See comment 5.

See comment 4.
Appendix II: Comments From the Department

of State

Page 37 GAO-02-41 State-Led Modernization Initiative

See comment 8.
Appendix II: Comments From the Department

of State

Page 38 GAO-02-41 State-Led Modernization Initiative

See comment 10.

See comment 9.
Appendix II: Comments From the Department

of State

Page 39 GAO-02-41 State-Led Modernization Initiative

See comment 11.
Appendix II: Comments From the Department

of State

Page 40 GAO-02-41 State-Led Modernization Initiative

See comment 12.
Appendix II: Comments From the Department

of State

Page 41 GAO-02-41 State-Led Modernization Initiative
Appendix II: Comments From the Department

of State

Page 42 GAO-02-41 State-Led Modernization Initiative
Appendix II: Comments From the Department

of State

Page 43 GAO-02-41 State-Led Modernization Initiative
Appendix II: Comments From the Department

of State

Page 44 GAO-02-41 State-Led Modernization Initiative

1. See comments 2 through 12.

2. We did not address the adequacy of funding for near-term activities as

part of this review.

3. We disagree. Our report clearly links our findings, conclusions, and

recommendations. Specifically, the report states that the level of

management rigor and discipline needed to effectively manage prototype

and pilot-testing activities is not as demanding as that needed for

acquisition and deployment of a system that is to be used operationally.

For example, while State’s lack of an enterprise architecture to guide and

constrain prototype and pilot-testing activities does not introduce

significant risk, it would introduce considerable risk for State to begin the

next phase of the program without this architecture. Accordingly, we

concluded that State’s existing management controls were not adequate

for the next phase of the program, and we made recommendations that are

intended to proactively provide a framework for effectively managing the

more demanding next phase of the program. We made no

recommendations intended to correct problems with State’s management

of its near-term activities because we concluded that controls over the less

demanding prototype and pilot-testing activities were adequate.

4. We disagree. Our report clearly describes how State’s existing

management controls are not adequate for the next phase of the program.

For example, State does not yet have an enterprise architecture to

effectively guide and constrain investment in the system. Our experience

with federal agencies has shown that attempting to define and build major

systems without first completing and using an enterprise architecture

often results in systems that are duplicative, not well integrated,

unnecessarily costly to maintain and interface, and not effectively

optimizing mission performance. Similarly, State has not yet implemented

an effective process for managing requirements for an operational system,

as evidenced in the report by the fact that agency partners have not

formally approved requirements for the prototype and pilot systems.

Although not having formally approved requirements for building

prototypes and conducting pilot tests does not introduce significant risks,

it would introduce considerable risk for State to acquire a system that is to

be operationally deployed without having formally approved requirements.

5. We agree that State has not proposed program and management plans

beyond the prototype and pilot-testing activities.

GAO Comments
Appendix II: Comments From the Department

of State

Page 45 GAO-02-41 State-Led Modernization Initiative

6. We agree. State’s plans provide for preparing a prototype and pilot

testing evaluation report. Also, such a report will be needed by not only

the lead agency for acquiring and deploying the operational system, but

also all participating agencies that will need to integrate the operational

system with their respective legacy systems.

7. See comments 3, 4, and 5.

8. We agree. As stated in our report, the Secretary of State was directed to

lead a cabinet-level committee to implement the recommendations of the

Overseas Presence Advisory Panel, which included acquiring and

deploying a multiagency common platform knowledge management

system. While we recognize the possibility that State might not be the lead

agency for the next phase of the program, we strongly believe that, to

allow for program continuity and avoid unnecessary delays, these issues

must be addressed now. Since State is currently the lead agency for the

program, the recommendations are being directed to the Secretary of

State.

9. We agree with State’s comment, and note in our report, that the roles,

responsibilities, and accountabilities of State and its agency partners

would need to be fully and formally defined before moving beyond

planned near-term activities. Although State asserted during the course of

our review, and in its comments, that its present informal structure for

managing agency relationships is working and its comments assert that it

provided documents to us attesting to this on numerous occasions, this

documentation, while voluminous and repetitive, did not demonstrate

what the current process is and whether it was working. As a result, we

were forced to expand the scope of work to include a survey of State’s

agency partners to determine whether State had engaged them in programrelated activities.

From our survey, we were able to conclude that State had taken adequate

action to engage the agencies. Nevertheless, some agencies raised

concerns that must be addressed before beginning the next phase of the

program. For example, one agency partner stated that it had not

participated in an approval process for program planning documentation,

another partner stated that it was unaware of any approval process for

planning-related activities, and a third stated that it had not reviewed any

of the program planning documentation. Moreover, one agency partner

stated that a more formal program governance structure is needed,

especially for reviewing and approving such documents as the program’s

concept of operations and the system’s functional requirements. A similar
Appendix II: Comments From the Department

of State

Page 46 GAO-02-41 State-Led Modernization Initiative

statement was made in a meeting of the State-led Foreign Affairs Systems

Integration Architectural Working Group. Specifically, it was stated that

protocols needed to be established, such as officially listing the documents

that would be shared so that all participants would be aware of these

documents and have the opportunity to review them before making any

decisions about how to proceed. While these issues are less significant for

program near-term activities, they should be resolved before moving

beyond system prototyping and piloting activities. Accordingly, our report

contains proactive recommendations for doing so.

10. We agree with State’s comment that the need to formally define agency

roles, responsibilities, and accountabilities should be focused on the next

phase of the program. We disagree that there is a need to eliminate a

reference to insufficiency in defining existing roles, responsibilities, and

accountabilities regarding near-term program prototype and pilot-testing

activities because no such reference exists. In fact, our report concludes,

and State acknowledges in its comments, that management controls over

near-term activities are adequate.

11. See comments 3 and 4.

12. See comments 3, 4, and 5.
Appendix III: GAO Contact and Staff

Acknowledgment

Page 47 GAO-02-41 State-Led Modernization Initiative

Cynthia Jackson, (202) 512-5086

In addition to the person named above, Nabajyoti Barkakati, Katherine I.

Chu-Hickman, Lester P. Diamond, Patrick R. Dugan, Sophia Harrison, and

Kenneth A. Johnson made key contributions to this report.

Appendix III: GAO Contact and Staff

Acknowledgment

GAO Contact

Acknowledgments

(310204)
The General Accounting Office, the investigative arm of Congress, exists to

support Congress in meeting its constitutional responsibilities and to help

improve the performance and accountability of the federal government for the

American people. GAO examines the use of public funds; evaluates federal

programs and policies; and provides analyses, recommendations, and other

assistance to help Congress make informed oversight, policy, and funding

decisions. GAO’s commitment to good government is reflected in its core values

of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the

Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of

current reports and testimony and an expanding archive of older products. The

Web site features a search engine to help you locate documents using key words

and phrases. You can print these documents in their entirety, including charts and

other graphics.

Each day, GAO issues a list of newly released reports, testimony, and

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site

daily. The list contains links to the full-text document files. To have GAO E-mail

this list to you every afternoon, go to our home page and complete the easy-to-use

electronic order form found under “To Order GAO Products.”

The first copy of each printed report is free. Additional copies are $2 each. A

check or money order should be made out to the Superintendent of Documents.

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a

single address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

P.O. Box 37050

Washington, D.C. 20013

To order by phone: Voice: (202) 512-6000

TDD: (301) 413-0006

Fax: (202) 258-4066

GAO Building

Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)

Washington, D.C. 20013

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm,

E-mail: fraudnet@gao.gov, or

1-800-424-5454 (automated answering system).

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G. Street NW, Room 7149,

Washington, D.C. 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Mail or Phone

Visit GAO’s Document

Distribution Center

To Report Fraud,

Waste, and Abuse in

Federal Programs

Public Affairs