Financial Markets: Stronger System Controls and Oversight Needed to Prevent NASD Computer Outages
Released: 1994-12-21T00:00:00

Letter

Background

Established in 1939, NASD regulates (1) over-the-counter securities trading (that is, trading that does not occur on the floor of a stock exchange) and (2) all brokers and dealers conducting securities business with the public. NASD owns and operates NASDAQ, a computerized communication system that provides quotation information on and facilitates trade executions for 5,700 securities. Implemented in 1971, NASDAQ links a nationwide network of about 500 brokerage firms, called market makers. These firms maintain inventories of securities which they buy from or sell to investors.

During 1993, 66.5 billion shares of stock—totaling $1.35 trillion—were traded in this market. These volumes represent 43.6 percent of the total shares traded on U.S. stock markets, or about 32.6 percent of the total dollar value traded.

NASD’s headquarters is located in Washington, D.C. The Association’s automated quotation and trading systems are located in and operated from its primary data processing facility in Trumbull, Connecticut. Its backup systems are located at NASD’s data processing facility in Rockville, Maryland, which also houses automated administrative systems such as payroll, personnel, and market surveillance.

The U.S. securities markets are primarily governed by self-regulatory organizations, such as NASD, which, in turn, are overseen by SEC. While self-regulatory organizations are responsible for maintaining smooth and dependable operations with their automated systems, SEC is responsible for overseeing overall market operations, including systems used to support such operations.

Scope and Methodology

To determine the nature and causes of the outages and to better learn how NASD develops, tests, and operates systems, we interviewed NASD senior officials, including the Executive Vice President and Chief Technology Officer, the Senior Vice President for Production Services, the Vice President for Computer Operations, the Director for Quality Assurance, and the Director for Performance Measurement. In addition, these officials provided us with a minute-by-minute chronology of events as they occurred on July 14 and 15 and August 1. We also obtained and reviewed NASD’s policies and procedures for quality assurance and stress testing.

Finally, we visited the primary data processing facility in Trumbull, Connecticut, to observe the systems and the controls used to safeguard them.

We determined the impact of the recent NASD system outages on three categories of market participants—market makers, derivatives markets, and individual investors. For the market makers, we used a structured questionnaire to collect information from the top 12 market makers—Merrill Lynch; Smith Barney Shearson; Herzog, Heine, Geduld; Mayer & Schweitzer; Troster Singer Corporation; Goldman, Sachs & Company; Lehman Brothers; Morgan Stanley & Company; Bear, Stearns & Company; The First Boston Corporation; PaineWebber; and Sherwood Securities Corporation. Together, these 12 represent over 50 percent of NASD’s total trading volume.

Our questionnaire included inquiries on how the outages impacted the market makers’ ability to obtain information and execute trades, as well as questions on the impact of the outages on confidence in NASD systems and the market, and on future participation in this stock market. We met with six of the market makers and mailed the questionnaire to the others.

For the derivatives markets, we interviewed NASD officials including the Chief Operating Officer, Chief Technology Officer, and Chief Economist, as well as senior officials from the Chicago Board Options Exchange and the Chicago Mercantile Exchange.

Finally, to assess the impact of the outages on individual investors, we interviewed market makers, senior NASD officials, and officials from the National Association of Investors Corporation and the American Association of Individual Investors—representing about 440,000 members combined.

In assessing the adequacy of NASD’s plans to respond to contingencies and disasters, we conducted a walk-through of NASD’s backup facility in Rockville, Maryland. We also interviewed those NASD officials responsible for preparing, maintaining, and testing the Association’s contingency and disaster recovery plan. In addition, we reviewed NASD’s contingency and disaster recovery plan and processes, including examinations of the 1993-94 test schedules and results.

To determine how well NASD oversees its automated market systems and facilities, we examined the role of NASD’s Internal Review office and discussed the work it has done in the past and is planning to undertake in the future. We interviewed the Director of Internal Review and reviewed position descriptions for the auditors who review NASD’s systems and computer facilities. We also obtained and examined Internal Review’s audit plan detailing the scope of work to be performed through April 1995. Finally, we interviewed systems managers at both the primary and backup computing facilities to determine the extent of their involvement with Internal Review.

To determine how SEC generally oversees markets’ preparedness for contingencies and disasters, we interviewed senior officials in SEC’s Division of Market Regulation and obtained a chronology of events and supporting documentation regarding the Commission’s role and response to the NASD outages experienced during July 14 and 15 and August 1. In addition, we reviewed SEC’s automation review policy, the Commission’s report of its most recent inspection at NASD which occurred in 1992, and the audit report of the most recent review of automated NASD systems conducted by an independent public accountant in 1992.

We conducted our review from August through October 1994, in accordance with generally accepted government auditing standards. We discussed the contents of this report with senior officials from NASD and SEC’s Division of Market Regulation and incorporated their comments where appropriate.

System Outages Due to Software and Hardware Malfunctions

The system outages experienced by NASD in July and August were due to malfunctioning software and hardware. Specifically, on July 14, new communications software being implemented as part of NASD’s efforts to upgrade its system did not operate as intended and caused the system to fail. When NASD staff restarted the system, the communications software experienced additional problems.

Consequently, NASD shut down the system and reconfigured it to use the old communications software. NASD operated its system this way for the remainder of the trading day with only minor problems. In total, the outages caused the system to be down for about 14 minutes. According to NASD systems officials, they corrected one problem with the new communications software that evening. These officials also told us that they disabled a function of the new communications software that was causing a second problem that could not be fixed immediately, and reconfigured the system to use the old software for this function.

On July 15, before the normal market opening (9:30 a.m. EST), the system’s response time slowed to unacceptable levels while processing routine tasks, prompting NASD to delay opening the market. At about 11:00 a.m., NASD diagnosed the problem as a faulty hardware component (used to manage disk access and storage devices), took it off-line, and opened the market at 11:55 a.m., approximately 2-1/2 hours late. According to NASD systems officials, it took them about 1-1/2 hours after normal opening time to locate this problem because their focus was on the new software, while the problem was actually caused by an intermittent hardware failure.

After the market opened, NASD then opened a market function for exchange-listed securities—the Consolidated Quotation Service. Because this function had been closed for the morning, the transaction rate surged. Software controls in the communications software that were designed to manage (limit) the number of transactions the system would accept did not fully protect the system from this surge and transaction backlogs began to build, resulting in the system’s response time increasing. NASD responded by turning off selected automated services to reduce the processing workload. The system functioned with only minor problems for the rest of the trading day.

The outage on August 1, which lasted 34 minutes, was caused by a faulty circuit board in NASD’s backup electrical system. The backup electrical system, which consists of commercial-grade batteries and generators owned and operated by NASD, was activated because the power from the local utility company dipped to an unacceptable level. As designed, the backup battery system operated until the facility could be switched over to the backup generators; however, during the switchover, the circuit board responsible for monitoring the conversion malfunctioned. This resulted in a total loss of power to the data center. At this point, NASD switched operations to the backup data processing facility in Maryland and continued operations for the rest of the trading day.

To address this problem, NASD (1) replaced the circuit board, (2) had the contractor who supplied the backup electrical system determine why the board malfunctioned, (3) is considering purchasing a second backup electrical system of batteries, generators, and circuit board to supplement its existing backup electrical system, and (4) hired a contractor to assess other single points of failure in the backup electrical system.

Incomplete Testing May Have Prevented NASD From Detecting Software Malfunctions

The malfunctions that caused the July outages might have been avoided had NASD more thoroughly tested its software. Testing systems to assess their ability to operate as intended and process unusually large workloads—commonly referred to as quality assurance and stress testing, respectively—helps identify and correct system weaknesses before they cause data processing disruptions in a live operating environment.

NASD performs quality assurance and stress tests on its systems. For instance, quality assurance personnel test software to determine whether it meets established business requirements. However, NASD’s quality assurance testing was limited in scope. Specifically, quality assurance did not (1) test all requirements and (2) verify that the system would not operate in inappropriate ways. For example, one requirement of NASD’s communications software was to limit the total number of transactions the system could accept into its processing queue; however, quality assurance did not test this software function. On July 15 when NASD opened the market, the system accepted more transactions than it was designed to handle without having to re-queue transactions, which slowed system processing speed to unacceptable levels.

In addition, NASD has a Performance Measurement Unit responsible for stress testing. This unit tests systems to determine how they behave under high workloads and demanding conditions. However, these tests were also limited in scope. For example, NASD did not test the system with sufficient volume to drive the system beyond the point where it begins to re-queue transactions, nor with a heavy backlog of transactions, such as occurred on July 15.

NASD systems officials said that their quality assurance testing program is rigorous enough to catch most problems, but acknowledged that problems sometimes can go undetected. In addition, while these officials said that their stress testing is adequate, they also agreed that their stress tests could be expanded to include transaction backlog conditions similar to those experienced on July 15.

We also discussed with systems officials why they installed new communications software on Friday, July 15, a “double witching” day. On such Fridays, the market is potentially volatile because options and other related financial instruments expire and market participants may need to buy or sell stocks to meet obligations. Installing new software on such potentially volatile days increases the risk that system problems could worsen market conditions and therefore, should be avoided. SEC has also found this practice to be undesirable and has recommended since the July and August outages that NASD avoid installing new software on such days and that systems managers coordinate changes with top management at NASD headquarters who are knowledgeable about market conditions.

NASD systems officials told us that it is NASD’s policy not to install system changes on Fridays. While the installation of the new communications software carried over to July 15, due to the problems on July 14, these officials believed that there was minimal risk of a malfunction because (1) the software had been tested, (2) it was installed in a phased approach over a 2-week period, and (3) they were confident in the systems and the personnel who developed and operate the software. Additionally, the systems officials told us that they do discuss system changes with business managers at the primary site.

During the course of our work, NASD systems managers began to inform top business managers via electronic mail of all upcoming system changes and installation schedules. This notwithstanding, unless NASD strictly adheres to its policy of avoiding the installation of new software changes on potentially volatile days, it risks having system malfunctions exacerbate market conditions. This risk could be made greater by the fact that NASD will be making numerous changes as it upgrades its system.

Impact of Outages on Market Participants and Related Markets Varied

The recent outages experienced by NASD had varying effects on market participants and derivatives markets. For example, individual investors were not significantly affected and did not report complaints regarding the outages. Conversely, market makers were impacted because they did not have the benefit of NASD’s automated quotation and trading system to conduct business. In general, the impact was greatest on July 15 when the system was out for 2-1/2 hours. Nonetheless, 213 million shares—79 percent of the average daily volume for July—were traded that day.

Market Makers Unable to Perform Best and Efficient Trade Executions

Market makers surveyed characterized the impact of the outages as being very great because they could not obtain updated price quotations from NASD. Without this information, market makers were unable to facilitate the best and efficient execution of trades. Market makers stated that the July 15 outage was particularly severe because they could not get quote information for the first 2-1/2 hours. Their frustration was heightened by the fact that they were uncertain when the market would reopen.

However, to serve customers who were willing to buy or sell without updated quotes, market makers relied on other means, such as using broker-dealer owned trading systems (Instinet, for example) to execute trades. Market makers generally told us that the 2-1/2-hour delay of July 15 resulted in lost opportunities to do business. In addition, seven market makers stated that they lost revenue they collect for executing trades on a normal day due to the outage. Of these seven, three estimated that they lost 20 to 25 percent of such fees, while the remaining four firms had not estimated the extent of their monetary losses attributable to the outage.

Effect on Individual Investors Limited

According to securities industry officials we interviewed, the outages had little impact on individual investors, who hold about 55 percent of all NASD market stocks. First, officials from two nonprofit associations, representing about 440,000 individual investors, told us that while their members generally report events that affect them, no complaints were reported regarding the NASD outages. One official stated that individual investors tend to make long-term investments so that outages of 1 day would probably not affect them.

Second, according to NASD, the majority of individual investors who participate in the stock markets do so through mutual funds, which generally price their funds using end-of-day stock quotes. Since NASD provided end-of-day stock quotes on the days the system experienced outages, mutual funds, and thus most individual investors, were unaffected. Finally, one of the market makers whose business caters to individual investors told us that all of its trades were executed, although not immediately, given the unavailability of updated quotes to guarantee best price execution of trades.

Derivatives Market Trading Halted Without Quotes

The derivatives markets—such as the options and futures markets—trade products that derive their value from NASD and other markets’ stocks. The Chicago Board Options Exchange, which trades the largest number of NASD stock options, had to stop trading these instruments on July 15 and August 1 because quotation information, which is used to derive the price of options, was not available. In addition, according to options exchange officials, when NASD opened its market 2-1/2 hours later on July 15 and began transmitting quote information, the exchange encountered a large volume of orders that had to be processed in a relatively short time frame.

We also interviewed officials at the Chicago Mercantile Exchange, which trades the largest number of futures on stock indices whose values are derived from the value of stocks traded in NASD’s market. These officials stated that the outages had no discernable impact on their trading operations because the vendors who price the indices continued to do so using last available quote information from NASD.

While NASD Prepares for Contingencies, Management and System Control Weaknesses Exist

NASD has taken significant steps to prepare its systems for contingencies and disasters. It operates a backup computer facility to be used if there are problems or outages at the primary computer site. In addition, NASD has prepared a detailed plan that identifies critical operations and the key individuals responsible for carrying out specified procedures during emergencies, such as power outages and natural disasters. The Association also conducts tests to gauge staff preparedness.

However, there are management and control weaknesses in NASD’s contingency and disaster recovery activities. While these weaknesses did not contribute to the problems experienced on July 14 and 15 and August 1, they make NASD vulnerable to problems should emergencies occur. For example, the contingency and disaster recovery plan is incomplete and out of date. Certain contingency scenarios have not yet been drafted and incorporated into the plan and names of some emergency personnel, who are no longer in such positions, have not been updated. In addition, the plan does not clearly delineate who is responsible for making systems decisions during contingencies and disasters. During our limited scope review at the backup site, we also identified certain internal control weaknesses. For example, the data center is located over a storage room of paper products, posing a potential fire hazard.

During the course of our work, we brought these management and control weaknesses to NASD’s attention and they attributed the weaknesses to oversights on their part. NASD agreed to correct them immediately.

NASD Oversight of Systems Is Limited

We examined the role of NASD’s internal audit function in identifying system and control weaknesses. We found that despite NASD’s extensive reliance on automated systems to accomplish its mission, until recently it had only one auditor with computer expertise reviewing automated systems. In addition, the scope of the auditor’s work was generally limited to reviewing administrative systems located at the backup site.

According to NASD officials, while internal audit’s focus has been on administrative systems, it has performed some work on market-related systems. Specifically, internal audit was involved during the development and implementation of the Fixed Income Pricing System and has reviewed Small Order Execution System outages. In addition, the internal audit function was established 2 years ago and is still in the process of establishing a program to ensure adequate audit coverage. Finally, internal audit focused its work on administrative systems at the backup site because these systems were judged to be more at risk than the market systems. This decision was based in part on the fact that the market systems had been reviewed by an external auditor in 1992 and internal audit believed it could rely on this work.

Because regular external and internal reviews are complementary management control practices used to oversee the use of automated systems, reviews by an external auditor are not a complete substitute for the day-to-day audit coverage provided by internal audit. Recognizing this, NASD officials stated that they (1) recently hired a second internal auditor with computer expertise, (2) plan to expand coverage of market systems in the audit work plan for the upcoming year, and (3) are discussing with SEC the frequency of external reviews.

SEC Has Issued Guidance on Contingency Preparation but Oversight Gaps Remain

Our past reviews of automated stock market systems have identified the need for SEC to establish the capability to address such technical issues as contingency and disaster recovery planning. SEC has subsequently taken steps to improve its oversight of the markets’ use of automation. For example, the Commission established an Office of Automation and International Markets and issued an automation review policy that encourages the securities markets to perform independent reviews of their automated systems and operations in such areas as contingency and disaster planning. The Commission planned to measure compliance with the policy by conducting inspections on a periodic basis.

However, gaps exist in SEC’s oversight program. First, it is unclear how often SEC expects the markets to perform independent automation reviews because the Commission’s policy does not state a specific frequency requirement. For example, the last such review at NASD was performed in 1992. In addition, SEC has not established how frequently it will perform its inspections. In practice, its inspections have been limited to about every 3 years because the Commission only has four technical staff members capable of conducting this work. Further, the scope of SEC’s inspections has been limited to reviewing other auditors’ work, rather than conducting first-hand reviews of system safeguards.

SEC also has not always followed up to ensure auditors’ recommendations are carried out. For example, in 1992, NASD had an external auditor review its systems. This auditor identified a serious control weakness in the way NASD modified software on the production system during emergencies. As part of its 1992 inspection, SEC reviewed the audit report, agreed with the auditor’s finding, and recommended that NASD take countermeasures to mitigate this weakness. However, at the time of our work, NASD had not yet corrected this weakness. While SEC officials told us that it is their goal to discuss all unresolved audit findings with NASD and the other markets during periodic briefings on market automation developments, these officials acknowledged that they had not taken action to ensure NASD had implemented the recommendation.

Officials in SEC’s Market Regulation Division stated that the automation review policy is still evolving and for this reason, they have not yet finalized all of its requirements. For example, SEC staff are currently negotiating with the securities markets to determine how often the external reviews will be performed and expect to reach agreement soon. In addition, the Market Regulation officials advised us that they are now including first-hand reviews of selected system safeguards as part of their inspections.

However, these officials told us that they are unclear what the optimal frequency for inspections should be. They also told us that SEC would be unable to conduct more frequent inspections and other oversight activities because it only has four computer specialists to oversee market automation at over 19 markets and other related organizations, such as clearing agencies and depositories. In addition, they said that hiring additional staff has been deferred by the Division due to other priorities. On November 17, 1994, these officials told us that they had recently received authority to hire two additional technical staff and were in the process of advertising the positions. Until SEC determines the appropriate frequency of inspections, it cannot be sure it has the correct number of technical staff to oversee automated market systems.

Conclusions

NASD is aware of the importance of maintaining reliable systems and providing backup in the case of emergencies, and is taking action to correct the weaknesses identified in this report. Addressing these weaknesses will lower the risk of future outages and enable NASD to respond more quickly and appropriately to future contingencies and disasters.

While SEC has made progress in strengthening oversight of market automation, gaps still exist in its oversight program. Until SEC fills these gaps, the Commission cannot ensure that it is adequately overseeing the rapid growth of automation in the securities industry.

Recommendations

We recommend that the Chairman, SEC, ensure that NASD expands testing processes for its market systems to better detect performs a thorough assessment of its existing systems environment to avoids implementing software changes on potentially volatile trading days; corrects weaknesses in its contingency and disaster recovery plan and backup data processing facility; and regularly schedules and conducts audits of its market systems.

In addition, we recommend that SEC’s Chairman (1) reach agreement with securities markets on the frequency of independent reviews, (2) determine SEC inspection frequency needed to ensure adequate oversight of market systems and facilities, and (3) follow up on systems auditors’ recommendations and ensure that the recommendations are adequately resolved. Given that the gaps in Commission oversight are attributable in part to a lack of technical staff, the Chairman should also determine the number of staff needed to adequately oversee the rapid growth of market automation and report this information to the Commission’s congressional appropriations and authorization committees in time for consideration in next year’s budget.

Agency Comments and Our Evaluation

We discussed the contents of this report with senior officials from NASD and SEC’s Division of Market Regulation. We incorporated their comments where appropriate. SEC officials agreed with our findings, conclusions, and recommendations. Except as noted below, NASD officials also agreed with the report.

NASD officials disagreed with our characterization that their software testing approach is limited or incomplete in scope. They said that they have adopted a rigorous approach to testing. In addition, NASD said that while its approach may differ from other approaches, it is successful, as demonstrated by the significant number of changes that have been introduced over the years without problems. Nevertheless, NASD officials told us that as an act of caution, they will engage an independent reviewer to assess the testing function and will respond appropriately to the reviewer’s recommendations. We believe that this is a prudent step.

We are sending copies of this report to interested congressional committees, the Chairman of the Securities and Exchange Commission, the President and Chief Executive Officer of the National Association of Securities Dealers, and to other interested parties. Copies will also be made available to others upon request. Please call me at (202) 512-6418 if you or your staffs have questions about this report. Other major contributors are listed in appendix I.

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Linda D. Koontz, Associate Director Gary N. Mountjoy, Senior Evaluator William D. Hadesty, Technical Assistant Director Kevin G. McCarthy, Senior Evaluator Sabine R. Paul, Senior Information Systems Analyst The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.