FDA Import Automation: Serious Management and Systems Development Problems Persist
Released: 1995-10-30T00:00:00

Letter

Background

FDA’s overall mission is to protect the public from selected domestic or imported foods, drugs, cosmetics, biological products, and medical devices and from products that make fraudulent or misleading claims that might threaten public health and safety. On matters relating to its import operations, FDA’s Office of Regulatory Affairs provides guidance and systems support, and performs planning, budgeting, and reporting activities for 6 regional offices, 21 district offices, and about 130 resident inspection posts. Imported products can enter the United States at seaports, airports, courier hubs, and border crossings. The volume of import entries subject to FDA regulations has been increasing over the last 20 years from about 590,000 entries in 1975 to about 1.6 million entries currently, and is expected to reach 2 million entries by 2000.

Products imported into the United States must be cleared first by Customs, whose responsibilities include assessing and collecting revenues from imports, enforcing customs and related laws, and assisting in the administration and enforcement of other provisions of laws and regulations on behalf of 60 federal agencies. Import brokers act as agents for importers and process the information required to bring products into the United States. Brokers can electronically transmit data on their products to Customs through an automated interface with Customs’ Automated Commercial System (ACS). If Customs determines that a product requires FDA approval before being released into the domestic market, such as for regulated food and drugs, the broker is to forward entry information to FDA for review.

Under FDA’s manual entry and review process, brokers must submit entry documents (an FDA-701, invoice, and associated certifications) to FDA for each shipment. Using these documents, FDA inspectors at the port of entry decide whether to release the shipment for entry, examine the shipment by inspecting it there, perform paper or laboratory examination of it for possible refusal due to violations of laws and regulations, or detain it until the broker furnishes additional information. Entry documents can range from a few pages to as many as 40 pages depending on the type and volume of goods in a shipment. The time interval between when the broker submits the documents to FDA and when the broker receives a release or examination decision from FDA averages 2 days.

As the volume of imports continued to grow, FDA recognized a need to automate and expedite its entry and review process. Also, FDA envisioned that an automated system would provide a method to capture and share historical data to bring uniformity to its enforcement decisions for detecting and preventing “port shopping” by importers. FDA found that because of its heavy workload or less interest in particular products at some ports, some importers tended to use the port of entry that provided them with the best opportunity for receiving FDA approval. In 1987, the FDA Commissioner formed a task force to develop a new automated system as recommended in a contractor-prepared feasibility study. This system, now known as OASIS, was intended to increase the efficiency and effectiveness of FDA’s program for monitoring imported products. In general, OASIS was expected to (1) increase the productivity of investigations personnel through automated interfaces with the laboratories, brokers and/or Customs, (2) improve screening of imports by providing suggestions for actions likely to result in discovery of violations, (3) provide faster turnaround for processing of importer’s entries and faster and more consistent responses, (4) provide national and district uniformity in processing of entries, and (5) maintain a base of information for generation of reports. OASIS was initially planned to be fully implemented in September 1989.

Scope and Methodology

We interviewed FDA and Customs officials in the Washington, D.C., area, and Seattle, Washington, to determine the operational objectives and timeframe for implementing OASIS. We also reviewed systems documentation provided by FDA, such as the system design, functional requirements, capacity analysis, risk assessment, regional contingency plan, implementation schedules, software support contracts, task orders, interagency agreement between FDA and Customs, and security and information resources management policies and procedures. We assessed FDA’s efforts to design, develop, and implement OASIS against GAO’s executive guide on the best practices of leading private and public organizations for strategic information management, and federal guidelines, such as the Federal Information Processing Standards Publications. In addition, we reviewed the 1994 joint self-assessment report on OASIS, the contractor’s cost-benefit analysis, and the System Design Review Committee’s report.

To monitor the implementation of OASIS, we visited and interviewed officials in FDA district offices and ports of entry in Seattle (system pilot location); Miami, Florida; Buffalo and New York, New York; and Detroit, Michigan. We also conducted telephone interviews with FDA officials in several other district offices. Interviews with FDA import managers and inspectors at these sites provided us with observations and examples of entries processed both manually and electronically at major FDA air, sea, and border ports. We also interviewed Customs officials and import brokers at the sites visited to obtain their perspectives on how the system has improved the import process.

Further, we interviewed officials at HHS, who were involved with the self-assessment and system design reviews. We also interviewed one of the prior software development contractors and the current contractor regarding their roles and responsibilities for the OASIS project.

We performed our work from April 1994 through June 1995, in accordance with generally accepted government auditing standards. We requested official comments on a draft of this report from the Secretary of Health and Human Services on August 10, 1995. As of September 18, 1995, we had not received any comments to include in the final version of this report.

Automated Import System Remains Incomplete

The development of OASIS is taking considerably longer than FDA officials expected. As shown in figure 1, after 8 years and three software development contractors, FDA still does not have a fully functional automated import system.

The original design, which was called the Import Support and Information System (ISIS), was for a large, nationwide, on-line, real-time, distributed FDA system. This system was modified following its 1991 pilot test and FDA’s agreement with Customs to include an automated interface with Customs’ ACS. As shown in figure 1, the modified system, known as OASIS, was pilot tested in Seattle, Washington, in 1992 and expanded to Portland, Oregon, and Blaine, Washington, in 1993.

OASIS, as implemented in the Seattle District locations, provides FDA inspectors the ability to (1) receive import entry data electronically from import brokers through interface with ACS, (2) receive results of preliminary processing against FDA’s selectivity criteria screening file, which is installed on ACS, that the shipment “May Proceed,” must be “Detained” for sampling, or must be held for “FDA Review,” (3) be alerted to potential problem areas with each line item of an entry, make follow-up screening decisions, and transmit these electronically to the broker through interface with ACS, (4) track actions taken and maintain historical data on all electronic import entries, and (5) eliminate many of the paper transactions among FDA, Customs, and import brokers. In addition, import brokers who interface with ACS receive preliminary and subsequent screening decisions relating to their electronic entries simultaneously with FDA.

However, software design problems experienced at the pilot locations made OASIS difficult to use. Such problems included slow response times when receiving and printing electronic data, moving from computer screen to computer screen, or going in and out of other systems while processing entries. These OASIS development problems prompted FDA to assemble a team of information resources management (IRM) representatives from HHS, PHS, and within FDA to pilot a self-assessment tool to analyze risks associated with the development of OASIS.

In June 1994, as a result of the exit briefing by the self-assessment team on its results and pending actions under way by FDA to replace the expiring OASIS contract, FDA’s Director of the Office of Information Resources Management called for the termination of both the development and deployment of all but the front-end portion of OASIS known as the electronic entry processing system or EEPS. FDA decided to maintain and refine OASIS in the Seattle District. In its July 1994 report, the self-assessment team concluded that the OASIS project was at high risk for system failure due to the lack of senior-level management involvement, project planning, and basic development processes as well as system design flaws, an insufficient budget, and a skeleton staff lacking adequate system design and implementation expertise.

In contrast to the OASIS functions described previously, EEPS allows FDA inspectors to receive the broker’s entry data from ACS, but only allows inspectors and brokers to receive the preliminary admissibility messages of either “May Proceed” or “FDA Review” for the entire entry. It is not capable of processing or transmitting any follow-up line-item decisions from FDA to the brokers. EEPS was deployed to 114 ports between March 1994 and June 1995, with 103 additional ports expected to be automated by the end of 1995.

According to import brokers and FDA inspectors we interviewed, even EEPS’ limited capability provides them quicker notifications as to the admissibility of imported shipments and reduces the amount of paperwork required from brokers. For “May Proceed” decisions, paper entry documentation is generally eliminated. For example, during the month of June 1995, FDA reported that of 2,520 brokers who interfaced with Customs’ ACS in the Seattle District and EEPS ports, 1,585, or 63 percent, used electronic filing for 178,412 FDA entries, and that 78 percent of the 1,585 electronic filers were not required to submit entry documentation for “May Proceed” entries. Table 1 below compares the traditional manual entry process to EEPS.

As recommended in 1994 by the self-assessment team, HHS, PHS, and FDA formed a systems design review committee to determine if (1) the OASIS design adequately meets the user requirements, (2) FDA computer hardware, or platform, is adequate for the system, (3) real-time access is necessary, and (4) telecommunications are adequate. The committee’s June 1995 report addressed the first three items. FDA’s telecommunications management branch is conducting an agencywide study on the telecommunications and network capacity and capabilities needed. The results of this study are expected in February 1996.

The committee’s June 1995 report stated that (1) the OASIS system design contains significant deficiencies, (2) the adequacy of the agency platform cannot be determined because certain stress and system load tests have not been performed or documented, and (3) real-time access is not necessary.

Serious Management and Systems Development Problems Persist

Although further development and deployment of the OASIS system is on hold, completion of a successful import system remains a major information resource management goal for FDA. We previously reportedon FDA’s need to address systems development problems and implementation delays, and our current review identified many of the same problems reported by the self-assessment team. In addition, we found that the OASIS project lacked necessary cost and performance information and did not consider some proven best practices of leading organizations that help ensure successful systems development. These problems must be resolved if FDA is to complete its automation of import operations.

Beginning in late September 1994 with the award of a new agencywide strategic information systems support contract, FDA began to address some of the systems development process problems identified but continued to lack effective senior-level management and direction, as well as a systems project management team with information technology expertise. In addition, FDA has made little progress in implementing basic systems development procedures, including conducting user acceptance testing and a risk assessment. Recent developments include the completion of a system design review which concluded that OASIS was not ready for national implementation and recommended an immediate reengineering effort.

Oversight and Project Management Expertise Are Critical

FDA top management did not adequately oversee the OASIS project and did not provide clear direction and appropriate resources needed to support the project. We found that this situation was largely due to an IRM structure that did not clearly define control and lines of accountability for the OASIS project. In addition, we found that the OASIS project was directed by managers who lacked the systems development training and expertise to successfully design, develop, deploy, and maintain an information system.

Management Oversight

The Deputy Commissioner for the Office of Management and Systems is both the chief financial officer and the senior IRM official for FDA. IRM activities on the OASIS project are the responsibility of the Office of Information Resources Management (OIRM) under the Deputy Commissioner for the Office of Management and Systems and the Office of Regulatory Affairs (ORA) under the Deputy Commissioner for the Office of Operations. As shown in figure 2, many FDA offices and divisions have some involvement with the OASIS project.

The responsibilities of OIRM include (1) ensuring that the agency’s 5-year strategic plan for acquisition and development of information resources is prepared and implemented, (2) ensuring that the most cost-effective approach is applied when acquiring information technology, and (3) approving acquisitions and ensuring that IRM goals and strategies are achieved.

Since the OASIS project began, its planning, design, development, implementation, and contractor acquisition and interaction resided primarily within the divisions of import operations and information systems in ORA. However, ORA’s requests for procurement authority for OASIS were and continue to be reviewed and approved by OIRM. The Deputy Commissioner for Management and Systems told us that since the award of the current strategic information systems contract in September 1994, the Associate Commissioner for OIRM has been charged with providing ORA with continuous technical consultation and scrutiny of all contractor task orders and deliverables prepared under ORA’s direction. The OASIS project manager is the director of the strategic initiatives staff, which is part of ORA, and does not report directly to OIRM officials.

In addition, some oversight has been provided at the department level. In accordance with the Paperwork Reduction Act, as amended, the HHS Secretary designated a senior official who is responsible for ensuring agency compliance with and prompt, efficient, and effective implementation of the information policies and IRM responsibilities under the Act. The designated senior official at HHS is the Assistant Secretary for Management and Budget, who has delegated certain authorities—such as for procurement—to agencies within the Public Health Service, including FDA. The HHS Deputy Assistant Secretary for IRM, who reports directly to the designated senior official, is responsible for management and operation of the department’s IRM program. It is at this level that HHS has provided FDA with assistance on both the self-assessment team and system design review committee.

The joint FDA/PHS/HHS self-assessment report indicated that FDA senior-level management needed to be closely involved with OASIS due to the visibility of the system and the troubled system development history. The report stated that the Commissioner or other top FDA officials did not receive regularly scheduled progress reports on the project. We found several memoranda dating back to 1989 in which OIRM raised concerns to ORA about the cost, complexity, and lack of well-defined requirements, alternatives, and planning regarding OASIS. Nonetheless, the project continued under the direction of ORA until June 1994, when the OIRM director called for the termination of further development based upon the results of the self-assessment team.

It is critical that senior-level oversight of this automation effort be established to ensure that information technology is acquired, used, and managed to improve the performance of FDA’s public health and safety mission, and that responsibility and accountability are improved. As discussed in GAO’s May 1994 publication on the best practices of leading private and public organizations for strategic information management, these organizations have found that without senior executives recognizing the value of improving information management, meaningful change is slow and sometimes nearly impossible.

Project Management

As discussed above, ORA administered the day-to-day management of the OASIS project. We found, however, that ORA did not have the systems development expertise in-house to perform these functions. Our review of the experience and qualification statements of OASIS project management showed that the ORA Deputy Associate Commissioner—the senior project official, the project manager, and the project officer did not have any systems development training or experience. The OASIS project manager concurred with our finding in a February 1995 memorandum, which stated that ORA did not have employees with adequate knowledge and experience in life-cycle methodology and related skills, all of which were important to a system of OASIS’ complexity. The memorandum stated that ORA planned to use its current software development and support contractor to address this deficiency in systems development and hardware acquisition expertise.

During our review, the self-assessment team recommended in its July 1994 report that ORA request and accept assistance from another FDA component, PHS, or HHS to address deficiencies in staff knowledge. As stated previously, ORA receives oversight from OIRM for task order review and approval, but not day-to-day assistance from this or other sources as recommended. ORA still does not have someone with the system development expertise to oversee the OASIS project and monitor the contractor’s work.

A best practice that can lead to improved mission performance is to ensure that skills and knowledge of line and information management professionals are upgraded. Also useful is establishing customer/supplier relationships internally and defining roles between line managers and information management support professionals to maximize management processes. Lastly, the chance of a breakdown between the agency and contractors is great when the agency does not have information management professionals with the needed expertise to assist line management in evaluating and supervising contractor performance.

Cost and Performance Information Are Needed

We found that FDA has not been effective in controlling costs or monitoring the progress of OASIS. FDA officials informed us that they did not have a cost accounting system that would enable them to clearly identify the costs of the OASIS project. They said that some of this cost was commingled with other information systems projects. For example, despite our repeated attempts to obtain the systems life-cycle cost for OASIS from its inception through the current fiscal year, FDA did not provide us with cost data until July 1995. This information was prepared by FDA’s contractor and submitted in June 1995, as part of a cost-benefit analysis requested by FDA. According to information contained in the contractor’s report, the OASIS systems development costs were estimated to be $13.8 million from fiscal year 1987 through April 1995. We did not independently verify these estimates.

In addition, the agency did not properly account for or match OASIS costs with outcomes to determine if OASIS would meet FDA’s needs within its budget allocation. Accurate accounting of all project costs will be crucial since FDA is supportive of legislation that would allow the agency to collect user fees for imports processed through the automated system to offset the costs of developing, deploying and supporting the system. Also, the importance of an import screening system to FDA’s operations and the import community warrants the maintenance of reliable cost and performance information to keep congressional appropriations and oversight committees informed of the status of any systems development effort.

ORA officials we interviewed told us that they did not establish any baseline measures to assess current and expected OASIS operational and technical performance. As discussed in GAO’s best practices publication, standard performance measurement practices focus on benefits, costs, and risks and, in most cases, include program outcomes, resource consumption, and elapsed time (cycle time) of specific work processes, activities, or transactions. Performance measures act as a common focus, allowing management to target problem areas, highlight successes, and generally increase the rate of performance improvement through enhanced learning. Such measures would allow top management to assess and manage the risk associated with its import automation effort, and to control the trade-offs between continued funding of existing operations and developing new performance capabilities.

Basic Systems Development Procedures Not Followed

We found that FDA did not follow sound systems development procedures, such as those outlined in federal guidelines, when developing OASIS because its project management team lacked expertise and training in systems development. Specifically, FDA did not (1) validate its criteria for electronically screening import entries, (2) conduct user acceptance testing, (3) conduct a risk assessment or prepare a security plan to address contingencies or backup procedures to be used in the event of disasters or threats to FDA’s computer facilities, equipment, and data, and (4) conduct a cost-benefit analysis. Many of these problems were brought to FDA’s attention as early as 1988. The following systems development problems must be resolved if FDA is to avoid continued criticism of its attempts to complete an automated import system.

No FDA validation of screening criteria. FDA had not validated the import admissibility screening criteria that reside in Customs’ ACS. Validation is essential to ensure that import entries are processed accurately and that potentially unsafe products are properly identified for “FDA Review.” OASIS project officials in ORA said that they did not have access to the criteria in ACS and could only validate information contained in the ACS-generated error reports. Moreover, these officials stated that they did not know if Customs corrected all the errors they identified. The joint self-assessment report also concluded that FDA did not have an adequate verification and validation process for its software and documentation.

Did not conduct user acceptance testing. The self-assessment report stated that FDA did not have written acceptance criteria or test plans. For example, FDA did not conduct nor participate with Customs in user acceptance testing prior or subsequent to implementing the ACS interface. ORA’s Deputy Associate Commissioner told us that it relied on and trusted Customs to ensure that the screening criteria database was functioning as intended.

Security plan not developed. Until recently, FDA had not conducted a risk assessment or developed a disaster recovery plan for EEPS, as required by federal guidelines. In 1992, FDA declared OASIS a “record system” subject to the requirements of the Privacy Act of 1974. Thereafter, FDA considered OASIS a critical-sensitive system. Also, the Computer Security Act of 1987 requires agencies to establish security plans and perform vulnerability assessments for all computer systems that contain sensitive information. In February 1995, FDA issued a risk assessment of EEPS at FDA headquarters and a contingency plan to address backup procedures for the Pacific Region, which runs the regional computer facility in the Seattle District office. However, we found that the risk assessment was incomplete and did not address major portions of EEPS. In addition, the contingency plan was not viable because FDA moved the OASIS processing function from Seattle to the larger processing facility in its headquarters in Rockville, Maryland. FDA does not have a contingency plan for ORA’s headquarters computer center. However, it plans to obtain a risk assessment of ORA’s information systems and contents through an interagency agreement with the Department of Transportation. As of May 1995, FDA could not tell us when a risk assessment and contingency plan would be performed at FDA headquarters to address security concerns for this mission-critical system.

No cost-benefit analysis conducted. At the beginning of our review, we learned that no one had performed a cost-benefit analysis for the OASIS project. This deficiency was also later reported by the self-assessment team. A cost-benefit analysis describes the development and operational costs of each alternative, and of nonrecurring (improved system operations and resource utilization) and recurring (operations and maintenance, including personnel) benefits that could be attained through the development of each proposed alternative. Such an analysis is useful to managers, users, and designers for analyzing alternative systems and will be essential to any decisions for further development of an automated import system.

ORA officials told us that they did not ask for such an analysis in the past. In February 1995, the current contractor was tasked with conducting a cost-benefit analysis, which was completed in June 1995. However, FDA did not request that the contractor perform an alternatives analysis. The current effort was limited to an analysis of OASIS’ historical costs from fiscal year 1987 through April 1995, which were estimated to be $13.8 million as well as projected costs from May 1995 through fiscal year 2001, which were estimated to be $26.2 million. The contractor also analyzed the costs and benefits of automation as compared to the current manual process.

Recent Developments

In June 1995, the System Design Review Committee issued its report on OASIS which concluded that the system is not ready for national implementation because of significant system deficiencies, including inconsistent user interface design and the lack of automated configuration management and version control. Consequently, the committee recommended that a reengineering effort begin immediately to design a system that would incorporate all customers’ needs, take advantage of modern technology and the strategic direction in which FDA is heading, and position FDA for the future.

In a July 10, 1995, meeting with FDA’s Deputy Commissioner for the Office of Management and Systems, we were told that FDA will not implement OASIS nationwide and will begin a reengineering effort. In addition, FDA agreed to the recommendations of the committee as stated in a July 12, 1995, correspondence from the Deputy Commissioner to ORA officials. However, the details of the reengineering effort have not yet been documented so it is not clear who will lead this effort, what will it involve, and how long will it take.

Reengineering is a formidable undertaking that requires an organization’s managers and employees to change the way they think and work. For example, after senior management recognizes the need for change and commits to reengineering, it then must direct the effort. Existing business processes should be described and analyzed, and measurable improvement goals should be set. In addition, senior management must also support the reengineering effort by identifying training needs and determining whether outside expertise is necessary. New business processes should then be designed and the organizational culture, structure, roles, and responsibilities should be changed to support these new processes. Finally, new business processes should be implemented by acquiring and installing new technology or redesigning existing technology to support the new processes.

FDA, though, has not yet clearly defined its reengineering effort and how it plans to link this effort to its information technology initiatives. This is critical if FDA is to achieve dramatic changes in overall performance and customer satisfaction.

Conclusions

A thorough understanding of the factors that led to FDA’s failure over the past 8 years to develop and implement an import system to meet its mission critical needs is crucial to help ensure that similar problems and obstacles are avoided in the future. As FDA plans its reengineering effort, it is presented with an opportunity to identify and correct its long-standing systems development problems. Because these problems can be attributed to a lack of top management oversight, systems expertise, and reliable cost and performance information, continued attention by FDA and HHS is vital to the success of this automation effort. It is crucial that FDA follow sound system development procedures, in conjunction with a well-defined reengineering strategy, if it is to successfully implement an import system and achieve its public health and safety mission.

Recommendations

We recommend that the Secretary of Health and Human Services direct the Assistant Secretary for Management and Budget and the Commissioner of the Food and Drug Administration to ensure that continuous top management oversight and systems expertise are provided to FDA as it proceeds with its import automation effort;

FDA develops and maintains reliable cost and performance information;

FDA follows sound systems development practices, including validating systems software, conducting user acceptance testing, developing a security plan, and conducting a cost-benefit analysis that includes an assessment of alternative systems.

We also recommend that the Secretary direct the Assistant Secretary and the Commissioner to clearly define how FDA plans to reengineer its import operations. At a minimum, FDA should (1) identify and analyze existing business processes and work flows, (2) obtain the necessary technical assistance and training to support its reengineering efforts, and (3) determine new information needs, application system requirements, and technology requirements necessary to support the new business processes.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 15 days from the date of this letter. We will then send copies of this report to the Secretary of Health and Human Services, the Commissioner of the Food and Drug Administration, the Director of the Office of Management and Budget, and other interested parties. Copies also will be made available to others upon request. This report was prepared under the direction of Patricia T. Taylor, Associate Director. You or your staff can reach me at (202) 512-6252, or Ms. Taylor at (202) 512-5539, if there are any questions on the report. Other major contributors are listed in appendix I.

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Seattle Regional Office

Susan T. Chin, Evaluator-in-Charge The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.