Year 2000 Computing Crisis: State Department Needs To Make Fundamental Improvements To Its Year 2000 Program
Released: 1998-08-28T00:00:00

Letter

Objectives, Scope, and Methodology

Our objectives were to assess (1) the overall status of State’s efforts to identify and correct its date-sensitive systems and (2) the appropriateness of State’s strategy and actions for remediating Year 2000 problems. In conducting our review, we assessed State’s Year 2000 efforts against our Year 2000 Assessment Guide. This guide addresses common issues affecting most federal agencies and presents a structured approach, as well as a checklist, to aid in planning, managing, and evaluating Year 2000 programs. This guidance describes five phases supported by program and project management activities. Each phase represents a major Year 2000 program activity or segment. The phases and a description of each follow.

Awareness - Define the Year 2000 problem and gain executive-level support and sponsorship for a Year 2000 program. Establish a Year 2000 program team and develop an overall strategy. Ensure that everyone in the organization is fully aware of the issue.

Assessment - Assess the Year 2000 impact on the enterprise. Identify core business areas and processes, inventory and analyze systems supporting the core business areas, and prioritize their conversion or replacement. Develop contingency plans to handle data exchange issues, lack of data, and bad data. Identify and secure the necessary resources.

Renovation - Convert, replace, or eliminate selected platforms, systems, databases, and utilities. Modify interfaces.

Validation - Test, verify, and validate converted or replaced platforms, systems, databases, and utilities. Test the performance, functionality, and integration of converted or replaced platforms, systems, databases, utilities, and interfaces in an environment that faithfully represents the operational environment.

Implementation - Implement converted or replaced platforms, systems, databases, utilities, and interfaces. Implement any and all contingency plans needed.

We also assessed State’s efforts against our Year 2000 Business Continuity and Contingency Planning Guide, which was issued as an exposure draft in March 1998. The guide provides a conceptual framework for helping large agencies manage the risk of potential Year 2000-induced disruptions to their operations. Like our Assessment Guide, it offers a structured approach for reviewing the adequacy of agency Year 2000 business continuity and contingency planning efforts.

To determine the overall status of State’s Year 2000 program, we analyzed the Department of State’s Year 2000 database, which includes data collected on a monthly basis from all of State’s bureaus, for four separate reporting periods: August 1997, December 1997, March 1998, and May 1998. State uses this database to track and measure program progress. We also reviewed the status reports State provided to the Office of Management and Budget (OMB) on a quarterly basis. To determine how State’s bureaus were implementing department policy and managing their Year 2000 program efforts, we interviewed Year 2000 coordinators at bureaus including Consular Affairs, Financial Management and Planning, Personnel, Diplomatic Security, and Information Management. We met with officials from the Diplomatic Telecommunications Service Program Office to determine what steps they were taking to ensure that telecommunications systems were Year 2000 compliant. We also reviewed internal State documents and reviews. We conducted our work from April 1997 through July 1998 in accordance with generally accepted government auditing standards.

We requested written comments on a draft of this report from the Secretary of State or her designee. The Acting Chief Financial Officer provided us with written comments that are discussed in the “Agency Comments and Our Evaluation” section and are reprinted in appendix I.

Background

Most of State’s automated information systems are vulnerable to the Year 2000 problem, which is rooted in the way dates are recorded and computed in automated information systems. For the past several decades, systems have typically used two digits to represent the year, such as “97” representing 1997, in order to conserve on electronic data storage and reduce operating costs. With this two-digit format, however, the Year 2000 is indistinguishable from 1900, or 2001 from 1901, etc.

In addition, any electronic device that contains a microprocessor or is dependent on a timing sequence may also be vulnerable to Year 2000 problems. This includes, but is not limited to, computer hardware, telecommunications equipment, building security systems, elevators, and medical equipment.

Should State fail to address the Year 2000 problem in time, its mission-critical operations could be severely degraded or disabled as the following examples illustrate.

The failure of State’s Consular Lookout and Security System (CLASS) would hinder the ability of overseas posts to effectively screen visa applicants who may have a criminal and/or terrorist background.

Embassy operations, such as property management and visa and passport processing, could be hindered at certain locations if State is unable to replace all of its noncompliant systems.

State’s messaging systems, which are critical to the effective conduct of diplomatic missions, could fail if telecommunications devices are not replaced or upgraded.

State has 262 systems comprising approximately 35 million lines of code written in over 17 programming languages. Major corporate systems include the Central Financial Management System (CFMS), the Central Personnel System (CPS), and CLASS. Through a strategy of system conversion and replacement, the department plans to remediate all of its noncompliant systems by March 31, 1999.

State supports its systems on a variety of hardware platforms, most of which are not Year 2000 compliant and will need to be fixed. Some of its corporate systems are operated on IBM mainframe computers at data processing centers in the Washington, D.C., area and overseas. According to State, some of its operating systems use antiquated “home grown” code and are presently not Year 2000 compliant. This environment is not stable, and State is currently working to resolve the issue.

The department also operates a variety of decentralized information technology platforms at posts around the world, including about 250 Wang VS minicomputers; 20,000 personal computers; and several hundred local area networks. Foreign service officers rely on this equipment for electronic mail, word processing, and other functions to develop reports and communicate information in support of State’s foreign policy objectives.

The Wang minicomputers will be replaced as part of State’s effort to modernize its information technology infrastructure. This project is known as A Logical Modernization Approach (ALMA). According to State’s IRM Tactical Plan, the ALMA project will (1) ensure that legacy Wang VS equipment and software is replaced by December 31, 1999, and (2) implement modern, open, and standards-based systems throughout the department. Under the direction of State’s Bureau of Information Resources Management, the department plans to deploy the ALMA infrastructure to all of State’s posts by the end of fiscal year 1999.

State’s Year 2000 Efforts to Date

State plans to resolve its Year 2000 problem using a phased process. In keeping with its decentralized approach to information technology management, State has charged its bureaus with responsibility for ensuring that all of their systems process dates correctly. Further, State is requiring the bureaus to redirect existing funds to correct their systems and will provide no additional funds for Year 2000 remediation. Although State estimated in its May 1998 quarterly report to OMB that it would cost $153 million to address its Year 2000 problem, in commenting on a draft of this report, the department stated that it is currently collecting and analyzing cost data and that an overall figure has not been finalized.

State’s Chief Information Officer (CIO) has overall responsibility for ensuring Year 2000 compliance. In addition, State has appointed a full-time Deputy CIO for Year 2000. The department also established a Year 2000 Steering Committee to (1) review new and ongoing information resources management (IRM) and non-IRM systems with regard to Year 2000 compliance, (2) conduct monthly reviews of Year 2000 efforts of all bureaus, and (3) reallocate resources across the department to meet Year 2000 needs as necessary. The Year 2000 Steering Committee is chaired by the Under Secretary for Management, and its membership includes the CIO, the Deputy CIO for Year 2000, the Chief Financial Officer, the Inspector General, the Assistant Secretaries of State for Diplomatic Security, Consular Affairs and Administration, and other senior officials. The CIO and the Year 2000 project manager monitor critical project implementation at key decision points and make specific recommendations to the Steering Committee. This committee meets monthly. Table 1 depicts the organizations involved in Year 2000 activities and their respective responsibilities.

To increase the awareness of Year 2000 problems and to foster coordination among components, State has taken the following actions.

In an April 1996 memo, the CIO alerted bureaus to the problem and called on them to attend a meeting to discuss the issue. In May 1996, State established a Year 2000 Project Office to manage the department’s Year 2000 program. In April 1997, the Year 2000 Project Office issued its Year 2000 Project Plan, which outlines the department’s strategy for achieving Year 2000 compliance. Subsequently, the project office distributed formal standards and guidance, including (1) a memorandum to all application developers (both in-house and contractor) providing guidance on Year 2000 data formats governing internal and external data exchange between information systems, (2) cable notices to all overseas posts informing them about the Year 2000 problem and identifying the steps they need to take to resolve the problem, and (3) Year 2000 planning and reporting guidance requiring bureaus to develop Year 2000 project plans and to provide quarterly (later changed to monthly) progress reports. In December 1997, State’s Year 2000 Project Office issued draft Year 2000 test planning and certification guidance to the department. This document describes the department’s Year 2000 test planning requirements, strategy, and schedule. In addition, the guidance identifies Year 2000 renovation test facilities for the IBM Mainframe, Wang, and PC/LAN test environments. In March 1998, State enlisted the Inspector General to help monitor its Year 2000 program, validate the data on Year 2000 status being reported by each component, identify problem areas, and recommend corrective actions. In March 1998, State reorganized the management of its Year 2000 effort. A Deputy CIO for Year 2000 was appointed as part of the general CIO office. The Under Secretary of State (Management) made each of the assistant secretaries personally responsible for ensuring that each of their bureaus is Year 2000 compliant.

Finally, an additional contractor, KPMG Peat Marwick LLP, was brought in to work alongside State personnel and the contractor already in place, Adsystech. KPMG Peat Marwick LLP was tasked with assisting in the overall management of the Year 2000 effort; Adsystech had been given responsibility for providing technical advice to bureaus for remediating systems. Adsystech is also responsible for collecting and analyzing data on the remediation process, and coordinating technical matters between State Department management and individual bureaus.

Using its assessment methods, State has identified a total of 262 systems, 64 mission critical and 198 nonmission critical. State has also determined that 40 mission-critical systems need to be remediated—27 of these need to be replaced and 13 need to be converted. In addition, State reports that 146 nonmission-critical systems need to be converted, replaced, or retired. Details of State’s assessment of its systems, as reported for May 1998, are shown in table 2.

Inadequate Progress in Remediating Systems

State’s progress in remediating systems has been inadequate. Of the 40 systems State has identified as mission-critical and is either converting or replacing, only 17 (about 42.5 percent) have completed renovation, 11 have completed validation, and only two have completed implementation. Tables 3 and 4 show the number of applications that have completed each phase along with the number of applications that have started but have not yet completed the phase.

In addition, the department has already conceded that it will not achieve its goal of eliminating all of its Wang software and hardware systems by the year 2000. As part of its IRM modernization program, State originally planned to eliminate all of its Wang VS systems (which include 21 mission-critical noncompliant systems) and begin running them on the Windows NT platform before January 1, 2000. According to State officials, however, because of delays in converting the Wang Systems to the Windows NT platform, the department will have to continue running some systems on the Wang platform after January 1, 2000. If all of the Wang systems cannot be replaced or made compliant before the year 2000, the department will not be able to run all of its mission-critical administrative applications overseas.

Further, a May 1998 report found that five of the mission-critical systems reported to OMB as compliant were, in fact, noncompliant and needed some form of additional remediation. The report also noted that 13 of all mission-critical systems were in a low degree of preparedness for certification and 8 systems were in a moderate degree of preparedness. In addition, seven of the mission-critical systems in a low degree of preparedness were scheduled to miss the OMB milestone date for implementation by 5 months, pushing their expected implementation to September 1999. These included systems essential to citizen services, such as immigrant and nonimmigrant visa issuance and tracking, and embassy and post security. One of these systems, the Immigrant Visa System, was reported to OMB as compliant. An additional system, the Non-Immigrant Visa System, was scheduled to miss the OMB milestone date for implementation by 1 month.

State Recognizes That a Mission-Based Perspective to Year 2000 Is Needed

As noted in our Assessment Guide and our Contingency Planning Guide, the Year 2000 problem is not just an information technology problem, but primarily a business problem. Thus, the process of identifying, ranking, and remediating information systems should include an identification of core business areas and business processes and assessments of the impact of information system failures on those business areas and processes. If this is not done, the agency will not have a good basis for prioritizing systems for correction or developing contingency plans that focus on the continuity of operations.

Until recently, State’s Year 2000 effort lacked a mission-based perspective. For example, at the time of our review, State had not determined its core business functions and linked these functions to its mission or to its support systems. In addition, the department had not conducted formal risk analyses of the majority of its systems. In responding to a draft of this report, State noted that it is currently developing a framework for a mission-based perspective for its Year 2000 problem. It has recently determined its core business functions and linked these functions to its mission. However, it has not yet linked its core business functions to support systems necessary to conduct these operations.

As further illustrated below, until it fully adopts this perspective, State will not be able to adequately prioritize its systems or develop meaningful contingency plans.

State Has Not Effectively Prioritized Systems for Correction

According to our Assessment Guide, an important aspect of the assessment phase is determining and prioritizing the correction of the systems that have the highest impact on an agency’s mission and thus need to be corrected first. This helps an agency ensure that its most vital systems are corrected before systems that do not support the agency’s core business.

State has provided its bureaus with a definition of priorities—routine, critical, and mission-critical—and charged them with the task of identifying and ranking their respective systems according to this definition. Mission critical, the highest priority, was defined as crucial to worldwide operations, affecting the public directly, or having national security implications. Subsequently, the bureaus assessed their respective systems and each provided the Year 2000 Project Office with a list of systems—64 in total—that they determined were mission-critical to department operations.

However, this process is flawed because it provides no means of distinguishing between individual bureaus’ priorities—some of which are essential to State’s core mission and some of which are not. For example, the following systems have been ranked by individual bureaus as mission critical:

REGIS, a system designed to register and track students who attend the

MSE Network, a system used to sort and track unclassified mail and

CLASS, a system designed to identify criminals and possible terrorists in order to block their entry into the United States;

CRIS, an on-line database used to track citizens involved in crises overseas; and ICARS, a system used for immigration control and reporting.

Clearly, CLASS, CRIS, and ICARS are much more important to State’s core missions than REGIS and MSE. But under State’s Year 2000 approach, they rank equally. Until State begins focusing on core business areas and processes, it will not have a basis for further ranking these systems for remediation.

Additionally, it appears that State has not placed enough priority on fixing its mission-critical systems before its nonmission-critical systems. In fact, as tables 3 and 4 indicate, State is making better progress on its nonmission-critical systems than on its mission-critical systems. For example, 31, or 21 percent, of nonmission-critical systems have reportedly completed the implementation phase, while only 2, or 5 percent, of mission-critical systems have done so.

State officials agree that the current prioritization process is flawed. In responding to a draft of this report, the department stated that it had recently identified its core business functions and planned to link them to the 64 systems previously identified as mission critical, thereby providing a functional basis for prioritizing their efforts. However, State did not plan to reassess the 198 systems previously identified as nonmission-critical using its new mission-based approach. Without reassessing all of its systems, State will not be able to fully ensure that the most critical functions will not be disrupted by the Year 2000 problem.

Business Continuity and Contingency Planning Is Inadequate

To mitigate the risk that Year 2000-related problems will disrupt operations, our guide on business continuity and contingency planning recommends that agencies perform risk assessments and develop realistic contingency plans during the assessment phase to ensure the continuity of critical operations and business processes. Contingency plans are vital because they identify the manual or other fallback procedures to be employed should systems miss their Year 2000 deadline or fail unexpectedly. These plans also define the specific conditions that will cause their activation.

State has directed its bureaus to develop written contingency plans for all mission-critical systems. At the time of our review, State reported that 16 written plans had been prepared, covering less than half of the 40 systems State identified as mission-critical and noncompliant. However, State was able to provide us with only six of these plans. These plans included only brief risk assessments and summary statements about possible alternate approaches for providing system functionality. They did not discuss the impact of the failure of system functionality on State’s mission.

Furthermore, State’s contingency planning is insufficient because it has not focused on ensuring the continuity of department operations and business processes. As noted in our Contingency Planning Guide, the risk of failure is not limited to an organization’s internal information systems. Many federal agencies also depend on information and data provided by their business partners—including other federal agencies, state and local agencies, international organizations, and private sector entities. In addition, they depend on services provided by the public infrastructure—including power, water, transportation, and voice and data telecommunications. Because of these risks, agencies must not limit their contingency planning effort to the risks posed by the Year 2000-induced failures on internal information systems. Rather, they must include the potential Year 2000 failures of others, including business partners and infrastructure service providers. By focusing only on its internal systems, State will not be able to protect itself against major disruptions of business operations.

In its May 1998 quarterly report to OMB on the status of its Year 2000 program, State acknowledged that its contingency planning efforts to date have focused on information technology systems rather than on the “larger picture of continuity of business operations.” To strengthen contingency planning, State has established a business continuity work group which includes members from the Year 2000 Steering Committee and is chaired by the Under Secretary for Management. This group is responsible for the development of business continuation strategies for Year 2000 risks. State has not identified a deadline for this group to complete its work.

State’s Management of Interfaces Has Been Ineffective

State systems interface with each other as well as with systems belonging to other federal agencies and international entities as shown in the following examples.

State’s central messaging system, which is used to transmit official diplomatic cables to overseas posts and other U.S. sites worldwide, interfaces with the Department of Defense.

State’s central personnel system interfaces with its payroll system to support payroll processing functions.

State’s CLASS system receives data on persons wanted for, or convicted of, drug-related crimes from the Drug Enforcement Agency’s (DEA) Lookout System.

As a result, it is essential that State ensure that all of its interfaces are Year 2000 compliant and that noncompliant interfacing partners will not introduce Year 2000-related errors into compliant State systems. Our Year 2000 Assessment Guide recommends that agreements with interface partners be initiated during the assessment phase to determine how and when interface conflicts will be resolved.

State has not managed the identification and correction of its interfaces effectively. First, it is still in the process of identifying its interfaces, even though our Year 2000 Assessment Guide recommended that this be done during the assessment phase. At the time of our review, State had identified 12 interfaces between mission-critical and external systems belonging to State and other agencies and organizations and 28 internal interfaces between bureaus that are affected by the Year 2000 problem. In addition, in June 1998, State reported to the President’s Council on Year 2000 Conversion that it maintained interfaces with commercial banks in 157 countries. According to State, 17 percent of its overseas accounts were Year 2000 compliant, 48 percent were scheduled to be compliant by December 1998, 7 percent in March 1999, 3 percent in June 1999, and 22 percent in December 1999. Three percent of the accounts were reported as having inadequate compliance plans.

However, State recently acknowledged that it could not identify every interface with other agencies or among the bureaus or verify whether all system owners were reporting on their interfaces or reporting correctly. State is now in the process of identifying these interfaces and verifying their progress.

Second, State has made little progress in developing agreements with its interface partners, which our Year 2000 Assessment Guide also recommended be done in the assessment phase in order to allow enough time for conflicts to be resolved. As of May 1998, State’s bureaus were reporting that Memorandums of Understanding had been completed for only 10 interfaces for systems that it has assessed as mission critical and noncompliant. Until it has agreements in place for the remaining interfaces, State will not have assurance that partners are working to correct interfaces effectively or in a timely manner.

Moreover, a May 27, 1998, report listed seven mission-critical systems as having a low degree of preparedness for Year 2000 certification based on the condition of their interfaces. The report also found problems with 20 other mission-critical systems due to interface problems.

Conclusions

The effective conduct of State operations hinges on its ability to successfully remediate its mission-critical computer systems before the Year 2000 deadline. While State has taken a number of actions to address this issue, its progress in several critical areas has been inadequate: only 17 of 40 systems that State has designated as mission-critical have completed renovation and it has not yet identified all of its interfaces. Further, if State continues its current approach, which lacks a mission-based perspective, it will risk spending time and resources fixing systems that have little bearing on its overall mission. It will also not be prepared to respond to unforeseen problems and delays.

Recommendations

We recommend that the Secretary of State ensure that senior program managers and the Chief Information Officer: (1) Reassess all of State’s systems using the new mission-based approach to identify those systems supporting the most critical business operations. (2) Ensure that systems identified as supporting critical business functions pursuant to recommendation 1 receive priority attention and resources over those systems that do not support critical business functions. (3) Redirect its contingency planning efforts to focus on the core business functions and supporting systems, particularly those supporting systems that are already scheduled to miss the OMB milestone date for implementation. (4) Ensure that the bureaus have identified and corrected interfaces and developed written memorandums of agreement with interface partners.

Agency Comments and Our Evaluation

State generally agreed with the conclusions and recommendations in our report. The department noted that it has already begun to respond to our observations and recommendations and that many of the specific concerns we raised have been independently identified by the department’s own consulting firm, KPMG Peat Marwick LLP. Additionally, State provided updated information about its management initiatives to address the Year 2000 problem, stating that it is rapidly implementing corrective measures for the problems cited in our report. While these changes demonstrate increased management awareness and attention to the Year 2000 problem, it will be critical for the department to follow through on these initiatives and ensure that they have a positive impact on the remediation, testing, and implementation of systems.

Furthermore, the department noted in its comments that it has recently identified its core business functions and linked these functions to its mission. The department also stated that it planned to link its core business functions to the 64 systems previously identified as mission critical. However, State did not plan to reevaluate the 198 systems previously identified as nonmission-critical. Until State applies its new mission-based perspective to all of its systems, it will not be able to fully ensure that the most critical functions will not be disrupted by the Year 2000 problem.

We are providing copies of this letter to the Ranking Minority Members of the Subcommittee on the Departments of Commerce, Justice, State, the Judiciary and Related Agencies, House Committee on Appropriations, and the House Committee on International Relations. We are also sending copies to the Chairmen and Ranking Minority Members of the Senate Special Committee on the Year 2000 Technology Problem, the Subcommittee on Commerce, Justice, and State, the Judiciary, and Related Agencies, Senate Committee on Appropriations, Senate Committee on Governmental Affairs, the Subcommittee on Government Management, Information and Technology, House Committee on Government Reform and Oversight, and the Subcommittee on Civil Service, House Committee on Government Reform and Oversight. We are also sending copies to the Secretary of State, the Director of the Office of Management and Budget, and other interested parties. Copies will be made available to others upon request.

If you have any questions on matters discussed in this report, please call me at (202) 512-6240. Major contributors to this report are listed in appendix II.

Comments From the Department of State

The following are GAO’s comments on the Department of State’s letter dated July 30, 1998.

GAO Comments

1. State’s detailed statistical information about its Year 2000 effort is constantly changing as State’s Year 2000 program evolves and remediation efforts progress. The information in our report represents the official figures reported to OMB in May 1998. The figures that State claims are more current are not substantially different from those reported in May 1998 and would not have any significant impact on our findings and recommendations. 2. Our assessment of the relative priority of fixing mission-critical and nonmission-critical systems did not include systems that State had designated as “compliant.” Instead, this assessment is based on the comparative number of noncompliant mission-critical and nonmission-critical systems that have completed the implementation phase. Only 2 (5 percent) of the 40 mission-critical noncompliant systems had been implemented as of May 1998 whereas 31 (21 percent) of the 146 nonmission-critical noncompliant systems had been implemented. We agree that systems currently considered “compliant” may not actually meet criteria for compliance and need to undergo their own, separate certification process. 3. State’s comments indicate that it is still taking a flawed approach to contingency planning. Like the prioritization of systems, contingency planning needs to be a top down rather than a bottom up process. That is, agencies must first identify their core business processes and assess the Year 2000 risk and impact of these processes. Subsequently, they can develop plans for each core business process and infrastructure component. As noted in our Year 2000 Contingency Planning Guide, this approach enables agencies to consider and mitigate risks that extend beyond individual applications or systems. For example, as noted in our report, State depends on information and data provided by other federal agencies, international organizations, and private sector entities. It also depends on services provided by the public infrastructure, including power, water, transportation, and voice and data telecommunications. Neither of these dependencies will be considered if contingency planning is focused on individual internal systems. 4. State provided no evidence of increased identification and awareness of commercial bank interfaces. Neither could the Department identify the number of international interfaces it might have. 5. In May 1998, State reported to OMB that its estimated cost to address its Year 2000 problem was $153 million. In our final report, we have noted that State no longer considers this figure to be accurate.

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

John Deferrari, Assistant Director Frank Deffer, Assistant Director Brian Spencer, Technical Adviser R.E. Canjar, Evaluator-In-Charge Cristina Chaplain, Communications Analyst The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.