0001

1 U.S. SECURITIES AND EXCHANGE COMMISSION 

2 

3 

4 

5 

6 ROUNDTABLE ON TECHNOLOGY AND TRADING: 

7 PROMOTING STABILITY IN TODAY'S MARKETS 

8 

9 

10 

11 

12 

13 

14 Tuesday, October 2, 2012 

15 10:00 a.m. 

16 

17 

18 

19 

20 

21 

22 

23 U.S. Securities and Exchange Commission 

24 100 F Street, N.E. 

25 Washington, D.C. 

0002

1 SEC COMMISSIONERS PRESENT: 

2 Hon. Mary Schapiro, Chairman 

3 Daniel Gallagher, Commissioner 

4 Troy Paredes, Commissioner 

5 Elisse Walter, Commissioner 

6 

7 PARTICIPANTS PRESENT: 

8 Sudhanshu Arya 

9 Gregg Berman 

10 Jim Burns 

11 Thomas Bayer 

12 Robert Cook 

13 Tom Eady 

14 Amy Edwards 

15 Robert Fishman 

16 Chris Isaacson 

17 Andrei Kirilenko 

18 Amar Kuchinad 

19 Dave Lauer 

20 Nancy Leveson 

21 Craig Lewis 

22 Lynne Markus 

23 Jamil Nazarali 

24 Lou Pastina 

25 Dawn Patterson 

0003

1 PARTICIPANTS PRESENT (Continued): 
2 Chris Rigg 

3 Jon Ross 

4 Todd Scharf 

5 David Shillman 

6 Heather Seidel 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

0004

1 C O N T E N T S 

2 PAGE 

3 Opening Statements: 

4 Robert Cook 5 

5 Hon. Mary Schapiro, Chairman, SEC 6 

6 

7 Dr. Nancy Leveson 36 

8 

9 Panel 1: Preventing Errors through Robust 

10 System Design, Deployment, 

11 and Operation 48 

12 

13 Panel 2: Responding to Errors and Malfunctions 

14 and Managing Crises in Real Time 107 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

0005

1 P R O C E E D I N G S 

2 MR. R. COOK: Good morning, everyone. My name 

3 is Robert Cook. I am the Director of the Division of 
4 Trading and Markets at the SEC, and it's my pleasure to 

5 welcome all of you here today to the Commission's 

6 Technology and Trading Roundtable. 

7 We're pleased you are all able to join us today 

8 for what we anticipate will be an informative discussion. 

9 I think we're going to be bringing in some more chairs in 

10 a moment for those of you standing in the back. So bear 

11 with us, but we're going to get the roundtable rolling in 

12 the meantime. 

13 First, let me just give a quick overview of the 

14 day. We'll have two panel discussions. This morning's 

15 panel will focus on error prevention. We will hear from 

16 technology experts about best practices and practical 

17 constraints for creating, deploying and operating mission 

18 critical systems, including those used to automatically 

19 generate and route orders, match trades, confirm 

20 transactions, and disseminate data. 

21 We expect to break from this panel at about 

22 12:15. We will then pick up again at around 2:00 p.m. 

23 for our afternoon panel, which will focus on error 

24 response. We expect to learn from our technology experts 

25 about how the markets might address erroneous quoting and 

0006

1 trading activity when it does occur in order to limit its 

2 impact, through independent filters, objective 

3 tests and other real time processes and crisis management 

4 procedures to detect, limit and possibly terminate the 

5 erroneous activity. 

6 Before we begin with our first panel, it's my 

7 honor to introduce Chairman Mary Schapiro and invite her 

8 to offer some opening remarks. 

9 CHAIRMAN SCHAPIRO: Thanks very much, Robert. 

10 I want to thank you and your team in Trading and Markets 

11 and in RiskFin for all the great work you've done 

12 organizing today's roundtable. 

13 Good morning, everyone, and I also want to 

14 thank all of our panelists for taking time to share your 

15 thoughts with us on market technology. And I also want 

16 to thank all of those who have already written in with 

17 comments. You've given us a number of very thoughtful 

18 recommendations that will become part of the record of 

19 what we're doing here today. 

20 To an extraordinary extent, the stability of 

21 our securities markets is tied to the technological 

22 infrastructure of those markets. As with virtually every 

23 industry, technology brings many benefits, and in that respect

24 our markets are no different. Thanks to technology, our 

25 securities markets are more efficient and accessible than 

0007

1 ever before. 

2 But we also know the technology has pitfalls, 

3 and when it doesn't work quite right, the consequences 

4 can be severe. Just imagine what can happen if an 

5 automated traffic light flashes green rather than red, if 
6 a wing flap on a plane goes up rather than down, if a 

7 railroad track switches and sends the train right rather 

8 than left. 

9 Similarly, there could be significant 

10 consequences for technological errors in our markets as 

11 well. Trading can be disrupted. Investors can suffer 

12 financial loss. Friends can be imperiled, and confidence 

13 in our markets broadly can erode. 

14 Today's roundtable will help us think through 

15 the issues and the steps we need to take to ensure that 

16 our markets remain the most robust, efficient and stable 

17 in the world. 

18 There are two basic concerns we need to focus 

19 on that are highly interrelated. These are, first, the 

20 structure of our markets, such as multiple execution 

21 venues, the presence of high frequency trading, dark 

22 pools and the like; and, second, the infrastructure of 

23 our markets, as in the technology that undergirds trading 

24 activity. 

25 To provide some perspective, in January of 

0008

1 2010, I asked the staff to begin a comprehensive review 

2 of the equity market structure. It was a review that 

3 included gathering views on everything from the impact of 

4 high frequency trading to the continued rise of dark 

5 pools, to the complexity of a multi-venue market system. 

6 The focus was not so much on the infrastructure of our 

7 markets, but on the way the markets and market 

8 participants operate and behave. 

9 Four months later when disorderly trading 

10 activities in the S&P E-Mini Markets spread to the 

11 equities market causing what is now known as the Flash 

12 Crash, we as an agency were well positioned to respond. 

13 Working with the exchanges, we quickly put in place a 

14 series of measures that have since helped to reduce the 

15 likelihood of another event like that from occurring. 

16 Within days we summoned the heads of every exchange to 

17 the SEC to hammer out common sense approaches to bolster 

18 our markets. And as a result of our efforts, we now have 

19 in place single stock circuit breakers to prevent stocks 

20 from falling too far too fast, and we have approved a 

21 more advanced limit up/limit down mechanism to limit 

22 excessive volatility. 

23 We now have in place a ban on stub quotes and 

24 rules clearly defining when a trade can be broken so as 

25 to help avoid circumstances that can lead to disorderly 

0009

1 trading. We now have in place rules banning naked access 

2 and requiring rigorous pre-trade risk controls designed 

3 to help mitigate disruptive trading at the source. And 

4 we now have rules requiring large traders, many of whom 

5 use high frequency trading strategies, to identify 

6 themselves so that the Commission can better monitor and 

7 analyze their trades, a process that other regulators 
8 overseas are beginning to emulate. 

9 Additionally, and perhaps most importantly, we 

10 have adopted a rule that requires SROs to develop plans 

11 for the first ever consolidated audit trail, a feature 

12 that will allow regulators to surveil and reconstruct 

13 trading across many platforms. 

14 But there are issues around market structure 

15 and the conduct of market participants that we should 

16 further examine, including the high volume of 

17 cancellations, a proliferation of order types, 

18 transparency, high frequency trading generally, 

19 potentially manipulative trading strategies, and data 

20 latencies for public investors, to name just a few. 

21 These issues will require attention, and we are committed 

22 to addressing them. 

23 Today's roundtable will focus more specifically 

24 on infrastructure not only because of its importance, but 

25 also because I do worry that this issue is at risk of 

0010

1 being lost and subsumed by the broader debates regarding 

2 market structure. After all, issues that get lost often 

3 do not get resolved, and these matters of infrastructure 

4 are essential to any holistic approach to improving how 

5 our markets operate. 

6 Consider for a moment the IPO of BATS on its 

7 own exchange and the IPO of Facebook on the NASDAQ 

8 exchange. Now, there are many views regarding the 

9 fragmented nature of simultaneous trading across multiple 

10 venues. I believe these IPO events evidence a very 

11 different set of concerns. Both events involved one of 

12 the few single exchange processes that remain in an 

13 otherwise fragmented market, namely, building a single 

14 order book and crossing trades at a single price to open 

15 trading for a newly public company. 

16 In the case of BATS, it was a flaw in new 

17 software code designed to conduct a corporate IPO 

18 auction. That mistake caused the matching engine for 

19 tickers in a certain range to enter into an infinite loop 

20 making these tickers, which included the symbol for BATS 

21 itself, inaccessible on BATS. 

22 In the case of NASDAQ, the IPO software was 

23 designed to accept cancellations submitted while the 

24 final IPO price, or the cross, is being calculated. 

25 Cancellations received during this time changed the order 

0011

1 book. By design the system recalculated the final IPO 

2 price to factor in the new state of the book, but again, 

3 changes were received before the system could print the 

4 opening trade, which resulted in additional 

5 recalculations. This condition persisted resulting in 

6 further delay of the opening print. 

7 These single exchange problems are not a result 

8 of complexity or fragmented markets, but rather a result 

9 of more basic Technology 101 issues. 
10 Consider as well the events this summer with 

11 Knight Capital, a trading firm that had just installed 

12 trading software that was intended to send orders to the 

13 NYSE's new retail liquidity program. Instead the 

14 software wound up sending a ton of orders into the 

15 market. As the market data that morning revealed, the 

16 software did not create patterns of rapid orders and 

17 cancels. Rather, the data showed a massive amount of 

18 orders resulting in executed trades that caused Knight 

19 Capital to accumulate significant and unwanted positions. 

20 This type of problem, as with the IPO mishaps, 

21 was again the result of basic Technology 101 issues. 

22 Events like these demonstrate the core infrastructure and 

23 technology issues that can be problematic in any market 

24 structure. However, though for today we are focusing on 

25 infrastructure issues, it is important to recognize how 

0012

1 the overall structure of our markets can affect how our 

2 infrastructure is designed and implemented. 

3 For example, we have a very competitive market 

4 environment in which rapid innovation and speed to market 

5 may compete with diligent testing and validation of the 

6 technologies that support such innovation. Our multi7 venue, interlinked market structure also means that an 

8 infrastructure failure by one part or one venue may 

9 cascade into other venues and affect many other parties. 

10 And, of course, the inherent speed of trading, 

11 which itself is partly a result of the competitive nature 

12 of our markets, means that even small, short-lived 

13 infrastructure issues can cause drastic harm. 

14 To be sure, several of the measures we have 

15 already approved have helped to strengthen our markets, 

16 even in the face of potential and inevitable 

17 technological errors. Indeed, several of the post Flash 

18 Crash reforms, such as the revisions to the clearly 

19 erroneous rules, helped limit the impact of the Knight 

20 Capital episode on other market participants. 

21 But limiting any harm resulting from 

22 technological errors is not as good as preventing the 

23 error in the first place, which is why we have instituted 

24 clear rules that require firms with access to our markets 

25 to have controls in place to reduce the chance of such 

0013

1 errors. 

2 But perhaps the strongest message from the 

3 Knight Capital episode is that the party committing an 

4 error may very well end up bearing a massive financial 

5 loss, and if there is a financial loss to be incurred, it 

6 is the firm committing the error that should suffer that 

7 loss, not its customers or other investors. That more 

8 than anything sends a wake-up call to the entire 

9 industry. 

10 Nonetheless, our concern is not whether a 

11 single firm might fail, but whether it causes collateral 
12 damage to investors and their confidence in the integrity 

13 and stability of our markets. So I'm very pleased the 

14 industry has been working overtime in the aftermath of 

15 the Knight Capital episode to address these issues, and 

16 I'm pleased our roundtable has spurred discussion. 

17 By focusing on the underlying nature of these 

18 incidents and hearing as we are today from experts in 

19 technology, I hope we can address these issues in an 

20 efficient, effective and expeditious manner. 

21 I will now turn the chair back to Robert Cook, 

22 Director of the Division of Trading and Markets, and his 

23 staff who will serve as moderators of today's discussion. 

24 Thank you. 

25 MR. R. COOK: Thank you, Chairman Schapiro. 

0014

1 Our panel for the morning session is entitled 

2 "Preventing Errors through Robust System Design, 

3 Deployment, and Operation." Before beginning this 

4 discussion, however, I want to introduce who we have 

5 around our square table here. 

6 First and foremost, it's my honor to 

7 acknowledge with appreciation that in addition to 

8 Chairman Schapiro, we're delighted to be joined by 

9 Commissioner Walter, Commissioner Paredes, and 

10 Commissioner Gallagher. 

11 In addition, there are a number of 

12 representatives of the Commission staff here who will be 

13 asking questions of our panelists. From the Division of 

14 Trading and Markets, Jim Burns and Gregg Berman will be 

15 assisting me in moderating the panel, and we're also 

16 joined by David Shillman, Heather Seidel, Todd Scharf, 

17 Amar Kuchinad, and Tom Eady. 

18 I should just note that Todd Scharf is acting 

19 head of our Automation Review Program, or ARP Program, 

20 and he's also the Commission's Chief Information Security 

21 Officer. 

22 From the Office of Information Technology we 

23 have Tom Bayer, who is the Commission's Chief Information 

24 Officer, and from the Division of Risk Strategy and 

25 Financial Innovation, we have Craig Lewis, the 

0015

1 Commission's Chief Economist, and Amy Edwards. And from 

2 our Office of Compliance, Inspections and Examinations we

3 have Robert Fishman this morning and I believe we'll have 

4 Dawn Patterson this afternoon. 

5 I should note for the record that any questions 

6 or observations that any of us on the staff might put 

7 forward today reflect our own views and do not 

8 necessarily reflect the views of the Commission, any 

9 Commissioner or our colleagues on the Commission staff. 

10 Of course, the folks we're all looking forward 

11 to hearing from the most are our distinguished 

12 panelists who we're delighted to have with us here today. 

13 They were invited to speak today because of their 
14 expertise and practical experience with the design, 

15 deployment and operation of complex technology systems. 

16 We very much appreciate their willingness to travel here 

17 and to share their observations and insights with us. 

18 I'm going to ask each of the panelists to 

19 briefly introduce themselves in a moment, but first I'd 

20 particularly like to extend a special welcome to two of 

21 our guests who come from academia because we've asked 

22 them to provide a brief presentation at the beginning of 

23 each of our panels to provide some broader context for 

24 the discussion. 

25 Sitting to my left and part of our first panel 

0016

1 is Dr. Nancy Leveson. Dr. Leveson is Professor of 

2 Aeronautics and Astronautics and also Professor of 

3 Engineering Systems at MIT. She is an elected member of 

4 the National Academy of Engineering. Professor Leveson 

5 conducts research on the topics of system safety, 

6 software safety, software and system engineering, and 

7 human-computer interaction. 

8 In 1999, Dr. Leveson received the ACM Allen 

9 Newell Award for outstanding computer science research, 

10 and in 1995, the AIA Information Systems Award for 

11 developing the field of software safety and for 

12 promoting responsible software and system engineering 

13 practices where life and property are at stake. 

14 Professor Leveson has kindly offered to provide 

15 the opening remarks for our first panel. 

16 And sitting to my right is Dr. Lynne Markus. 

17 Dr. Markus is the John W. Produska, Sr. Professor of 

18 Information and Process Management at Bentley University. 

19 She's also a research affiliate of MIT Sloan's Center for 

20 Information Systems Research. Professor Markus' teaching 

21 research and consulting interests include enterprise and 

22 interorganizational information systems, the unintended 

23 consequences of information technology and risk 

24 management strategies, and IT in the mortgage industry. 

25 She was named Fellow of the Association for 

0017

1 Information Systems in 2004 and received the AIS Leo 

2 Award for exceptional lifetime achievement in information 

3 systems in 2008. 

4 Professor Markus has kindly offered to 

5 participate on and provide opening remarks for our 

6 afternoon panel, but for this morning's panel she is 

7 sitting here with us to ask questions and provide 

8 observations as part of the discussion for the first 

9 panel. 

10 So I'd like to ask the rest of our panelists to 

11 briefly introduce themselves and the organizations that 

12 they represent, after which I'll ask Dr. Leveson to start 

13 us off with her prospective. So why don't we just go 

14 right on down the panel and people can introduce 

15 themselves? 
16 MR. ARYA: Good morning, Chairman Schapiro, 

17 Commissioners and division staff, and thank you for the 

18 opportunity to participate in today's discussion of 

19 technology in trading. My name is Sudhanshu Arya, and 

20 I'm a Managing Director at ITG. ITG is a global broker 

21 that executes as agent on behalf of institutional 

22 investors and broker-dealers both on and off exchange. 

23 We represent approximately three percent of daily volume 

24 in U.S. equities. 

25 At ITG I'm responsible for the full technology 

0018

1 cycle for liquidity management business. This includes 

2 all of our arithmetic trading, routing to exchanges of 

3 other trading venues, market office functions, and Posit, 

4 one of the largest, most established dark pools in the 

5 world. 

6 In our search for best execution, we're 

7 required to tackle a number of technical challenges, 

8 including track and climb positions, intermediating order 

9 routes in parent-child relationships, managing open field 

10 and canceled orders in the marketplace, and tackling 

11 latency, throughout and compliance demands associated 

12 with operating in today's environment. 

13 Our aspiration for error prevention is simple. 

14 We strive for zero errors in production. Although this 

15 goal may seem unrealistic, it establishes a culture of

16 prevention and planning. An error free production 

17 environment is key to our license to innovate. In an 

18 extremely competitive environment we are highly motivated 

19 to minimize trading risk and errors. 

20 ITG spends significant resources on prevention 

21 and recovery mechanisms that go above and beyond 

22 regulatory requirements. Today's roundtable discussion 

23 represents a good opportunity to enhance industry-wide 

24 efforts to prevent errors, but errors happen and always 

25 will. When they do occur, efficient recovery is 

0019

1 critical. Industry coordination based upon agreed upon 

2 metrics and established communications plans and 

3 protocols are crucial. 

4 We look forward to contributing to the 

5 industry's efforts to establish trading technology best 

6 practices, and we commend the division for organizing 

7 today's event. 

8 Thank you. 

9 MR. ISAACSON. Good morning. I'd like to thank 

10 Chairman Schapiro, fellow Commissioners and the SEC for 

11 arranging this roundtable on market technology and 

12 inviting me to participate. 

13 I'm a Chief Operating Officer of BATS Global 

14 Markets, a founding employee, and was an original 

15 developer of the BATS trading system. I oversee 

16 technology and operations at BATS. 

17 During my tenure at BATS, we've launched an ECN 
18 to the National Securities Exchanges, a European 

19 multilateral trading facility, a U.S. Options Exchange, 

20 and completed the acquisition of Chi-X Europe, and 

21 successfully upgraded the Chi-X Europe to the BATS 

22 technology. 

23 With the BATS team, I also responded to the 

24 unfortunate technical error that caused the withdrawal of 

25 our own IPO and have helped manage through the resulting 

0020

1 crisis. 

2 Financial markets, especially the U.S. equity 

3 market, have changed dramatically in the last 15 years. 

4 Numerous changes in regulation, such as the newer 

5 handling rules, Reg. ATS and Reg. NMS, along with mass 

6 adoption of electronic trading, have permanently 

7 transformed the markets. Well publicized events like May 

8 6th and recent technology problems have fueled concerns 

9 about the quality of market structure and overall market 

10 confidence. 

11 The technology problems this year have also 

12 raised concern about the level of control surrounding 

13 technology that drives our capital markets. While 

14 realizing the complex systems inevitably fail at times, 

15 today I look forward to discussing how we can better 

16 prevent these failures and minimize their impact on the 

17 broader market when they do occur. 

18 Regarding the prevention of errors, I'd like to 

19 discuss three key ideas today. First, better policies 

20 and procedures to ensure adequate time is devoted by 

21 market centers and participants to testing in different 

22 environment before production rollout. 

23 Second, the requirement that all market centers 

24 support test symbols that allow for testing of new 

25 trading strategies and order types in a live production 

0021

1 environment. 

2 Thirdly, more tools developed for and greater 

3 involvement of business and compliance personnel to 

4 enhance coverage of possible test case scenarios. 

5 Regarding the responding to errors and 

6 minimizing their impact, I suggest the following idea: 

7 Document the practice of remediation of known 

8 potential crisis scenarios; 

9 Enhance the monitoring to expedite 

10 investigation of unforeseen issues and their root causes. 

11 A primary example of enhanced monitoring should 

12 be the receipt of real time execution drop copies from 

13 market centers by participants in order to reconcile in 

14 real time the participant's record of executions versus 

15 the market center's record of executions. 

16 Thirdly, know who your key stakeholders are and 

17 be prepared as part of your crisis planning to reach out 

18 to them and keep them informed in as near real time as 

19 possible. 
20 And fourthly, and finally, kill switches at the 

21 exchanges ideally invoked in a coordinated fashion across 

22 the industry based on centralized, real time position 

23 information reported to the DTCC. 

24 I look forward to discussing many of these 

25 ideas among others with my fellow panelists and the SEC 

0022

1 today. 

2 MR. LAUER: Good morning, Chairman Schapiro, 

3 Commissioners and the SEC Division of Trading and 

4 Markets. Thank you very much for hosting this roundtable 

5 on such a critical issue, and thank you for having me. 

6 My name is Dave Lauer, and I'm currently consulting for 

7 Better Markets. 

8 At Better Markets we advocate reforms to help 

9 curb the abuses in the marketplace. I primarily focus on 

10 the abuses of unfettered technology, high frequency 

11 trading, and predatory trading practices. 

12 As well I'm consulting for IEX Group, which is 

13 a private company building an investor-owned and 

14 investor-focused equity market center. 

15 Since 2005, I have been studying the electronic 

16 markets, first, while building low latency trading 

17 infrastructure which was measured in microseconds, 

18 building actual hardware devices. That was at a start-up 

19 where we worked with most of the top high frequency 

20 trading desks and algorithmic execution desks. 

21 I then took a job after that as a quantitative 

22 analyst in charge of both research and trading both high 

23 and lower frequency trading strategies. 

24 After working for a time focused on both 

25 quantitative research as well as back-testing and trading 

0023

1 these strategies, I got substantial experience in 

2 understanding the complexities that high frequency 

3 trading strategies face in terms of order routing and the 

4 speed at which they had to move and make decisions. 

5 After witnessing the Flash Crash and many other 

6 smaller incidents on a near daily basis, I decided to 

7 leave the industry and try to use what I have learned to 

8 help try and fix some of the problems that we've all 

9 witnessed. 

10 I think many of us on the panel agree on some 

11 of the issues, such as enhanced testing and quality 

12 assurance are necessary. While many of the more mature 

13 firms in the industry do this, many of the less mature 

14 ones do not and often act with very much a technology 

15 start-up mentality. 

16 I believe technology has moved so quickly that 

17 most market participants have not been able to keep up. 

18 Maybe internally they have and they stay at the cutting 

19 edge, but very few firms at the cutting edge, high 

20 frequency trading firms included, understand the true 

21 complexities involved when all of these algorithms are 
22 interacting with each other in the marketplace and the 

23 nonlinear incidents that happen, such as the Flash Crash. 

24 I would urge the SEC to recognize that solving 

25 the problems of market structure, fairness and 

0024

1 orderliness or now technology issues as the technology is 

2 the driving force of the markets, and I would also urge 

3 the SEC to consider the tenets of the most disruptive 

4 technology force of our time, the Internet, which is 

5 based on openness, transparency, and a departure from the 

6 proprietary ways that have marked the technology 

7 revolution on Wall Street up until this point. 

8 I believe that with a true open, market-wide 

9 surveillance system we can confront many of the issues 

10 that we see by using and leveraging more advanced 

11 technology algorithms. 

12 Thank you. 

13 Good morning. I'd like to begin by extending 

14 my thanks to Chairman Schapiro, the Commission and staff 

15 for organizing this roundtable. 

16 My name is Jamil Nazarali, and I'm Senior 

17 Managing Director at Citadel Securities, a leading retail 

18 market maker in equities and listed equity options. 

19 I appreciate the opportunity to participate in 

20 this roundtable and to help advance the dialogue to 

21 ensure market stability and integrity in today's 

22 increasingly automated trading environment. 

23 I joined Citadel over a year ago and have been 

24 active in the industry throughout my career, most 

25 recently spending ten years at Knight Capital Group, 

0025

1 where I was responsible for electronic trading. 

2 Citadel Securities trades approximately 13 

3 percent of U.S. consolidated volume in equities and 21 

4 percent of U.S. listed equities volume. We operate an 

5 industry leading market making franchise and an 

6 institutional markets platform. Our team serves a 

7 diverse client base and is a major liquidity provider to 

8 retail investors in the U.S. and around the world. 

9 From these vantage points I've been able to see 

10 how for years automated trading systems have provided 

11 enormous benefits for everyday investors, dramatically 

12 lowering trading costs, improving market transparency, 

13 and increasing market efficiency. 

14 At Citadel, we have a vested interest in sound 

15 and stable markets. We are hopeful that today's 

16 discussion and follow-on work will help to ensure that

17 investors continue to benefit from automated trading. 

18 Thank you. 

19 MR. PASTINA: Good morning. I, too, would like 

20 to thank Chairman Schapiro, the SEC Commissioners and the 

21 staff for putting together today's roundtable to discuss 

22 some of the most pressing issues facing the U.S. capital 

23 markets today. 
24 My name is Lou Pastina, and I currently manage 

25 the NYSE Cash Equity Market Operations Group, including 

0026

1 the development schedule, the tactical operating plans, 

2 the NYSE trading floor, and the electronic trading 

3 component of the NYSE equity market. 

4 For the past 30 years, I have been involved in 

5 some capacity in nearly every aspect of the exchange's 

6 business. During this time frame, I've witnessed the 

7 evolution of the execution process in equity markets from 

8 the use of pencil and paper on the trading floor of the 

9 NYSE and NASDAQ's electronic quote screen phone system to 

10 a sophisticated web of servers that include several tiers 

11 of fully automated execution services via broker-dealer 

12 internalization, ATSes, ECNs, and exchanges. 

13 This web of technology has allowed for an 

14 unprecedented level of investor interaction that could, 

15 if fully realized, have a tremendously positive outcome 

16 for all investors. A component of maximizing this 

17 technology requires market participants providing 

18 execution services to implement a greater level of 

19 oversight of their development and implementation of 

20 these services which will provide consumers with a 

21 greater level of confidence in the U.S. equity market 

22 infrastructure. 

23 As you know, self-regulatory organizations 

24 operate under the direct purview of the Securities and 

25 Exchange Commission and participate in the SEC's 

0027

1 automated review policy program, as it pertains to the 

2 review of trading technology and security. The ARP 

3 Program conducts on-site annual reviews, quarterly 

4 updates calls, and engages in regular dialogue with us on 

5 matters relating to the technology development, quality 

6 assurance, and security functions of our exchanges. 

7 Although the program is voluntary, we have 

8 always treated it as mandatory and generally believe it 

9 serves several purposes, including assisting regulators 

10 in identifying issues that might be occurring across 

11 several SROs. 

12 As the industry and the SEC consider solutions 

13 that will enhance the stability of the capital markets, 

14 we believe it is important for any standards to be 

15 uniform across the industry and any additional 

16 requirements that are instituted should not exclusively 

17 place additional requirements on exchanges. 

18 Given the rapid rise in the number of trade 

19 execution venues and the technology used by traders to 

20 access each of those venues, it remains in the interest 

21 of all market participants to universalize those 

22 standards. Given the topic of this panel is preventing 

23 errors through robust system design, deployment and 

24 operation, there are two primary points I hope to outline 

25 today. 
0028

1 The first is the establishment of common 

2 standards that can be used in the industry for technology 

3 deployment. 

4 The second is a possible technology solution we 

5 believe will address a volume driven event similar to the 

6 one that occurred at Knight Securities. 

7 Everyone involved in the securities transaction 

8 business recognizes that there is no single way to go 

9 about delivering innovative technology options and 

10 solutions to market participants. This includes the 

11 differing functionalities available at venues as well as 

12 the deployment and testing of new technology introduced 

13 into the trading environment. 

14 However, given the complexity and 

15 interconnectedness of trading technologies that exist 

16 today, we believe the industry should consider 

17 establishing a common set of standards or best practices 

18 for all execution venues to follow regarding the 

19 implementation and deployment of trading technology. 

20 In particular, we believe that the industry 

21 could benefit from common standards for system 

22 development life cycles requiring minimum standards for 

23 specifications, coding, testing and implementation; 

24 employing capability models and certifying to minimum 

25 levels, perhaps making those metrics public; implementing 

0029

1 standard testing requirements for large scale 

2 deployments, including unit regression integration, 

3 customer-industry recovery and capacity testing, and 

4 having capacity and recovery standards. 

5 Consideration should be given to establishing 

6 minimum capacity standards and recovery standards for all 

7 execution venues. 

8 Establishing a common set of standards that are 

9 broadly accepted can go a long way to minimizing the 

10 confusion imbedded in an already cumbersome set of 

11 processes designed to eliminate problems from occurring 

12 before they're placed into production. 

13 Most of the existing market center controls 

14 today are focused on volatility in the market rather than 

15 volume, as was the case with the most recent major market 

16 technology issue. To that end the SEC implemented 

17 15c3-5, market access control rules from July 2011, which 

18 are designed to require broker-dealers to establish 

19 practices that should prevent market-wide issues as a 

20 result of excessive erroneous volumes. 

21 While it has yet to be determined if these 

22 rules are being implemented with the benefit they seek to 

23 provide, we believe that any additional rules that 

24 technology solutions discussed would be an added layer of 

25 protection for the market and not in lieu of 15c3-5. 

0030

1 As I mentioned, there are already several 
2 methodologies exchanges use to curtail trading 

3 disruptions. At the NYSE, we have three primary 

4 controls: throttling, liquidity replenishment points, 

5 and line monitoring. 

6 In addition, there are market-wide 

7 methodologies which include single stock circuit breakers

8 which will be replaced with limit up and limit down; 

9 market-wide circuit breakers; and clearly erroneous 

10 execution rules. Each of these items has worked as 

11 patches to address the problems they are designed to 

12 affect. However, none of them focus on excessive volumes 

13 from a single source. 

14 As you may know, subsequent to the issues at 

15 Knight, NYSE Euronext helped organize a working group of 

16 market participants from nearly every corner of the 

17 trading universe, including other SROs, broker-dealers, 

18 market makers, proprietary traders, and buy-side firms. 

19 One of the premises of the discussion was that although 

20 15c3 is designed to establish controls at the individual 

21 broker-dealer level, it does not protect the industry as 

22 a whole if the technology problem occurs at a 

23 broker-dealer regardless of their intent to comply with 

24 the rule. 

25 The working group recently submitted a comment 

0031

1 letter outlining a possible kill switch solution that we 

2 believe will provide a second layer of security should a 

3 volume based disruption occur again. As outlined in more 

4 detail in the letter, we believe a kill switch could be 

5 designed at the exchange level that would halt quoting 

6 and trading activity of a broker-dealer if it exceeded 

7 the established peak net notional volume threshold set by 

8 the broker-dealer. 

9 We believe that the broker-dealers would be 

10 best suited to choose what their peak net notional volume 

11 threshold should be within reasonable measures based on 

12 prior trading volumes. 

13 In addition, the exchange could include 

14 functionality that would automatically send an electronic 

15 message to the broker-dealer should it come within a 

16 certain percentage of its threshold, and unless the 

17 threshold is increased the broker-dealer's access to the 

18 change would halt upon reaching the threshold. 

19 While there are other specifics around such a 

20 proposal that must still be discussed, we believe that a 

21 kill switch-like solution may be the most widely accepted 

22 solution that can be implemented on a reasonably short 

23 time frame. 

24 In addition, we believe that there are several 

25 market participants that may have longer term solutions 

0032

1 to these issues as well. 

2 Thank you, again, for the opportunity to 

3 participate in today's discussions. We intend to 
4 continue our efforts to work with our colleagues in the 

5 industry to come to an appropriately measured solution 

6 and look forward to answering any questions. 

7 MR. RIGG: Good morning. First of all, I would 

8 like to thank Chairman Schapiro and the Commissioners and 

9 the staff. I very much appreciate the opportunity to 

10 participate in this panel. 

11 My name is Christopher Rigg. I'm from IBM's 

12 Global Business Services, which is our consulting 

13 business unit, and I lead our Application and Innovation 

14 Services Group focused on the banking and financial 

15 markets practice. 

16 That group is essentially the organization 

17 within IBM that focuses on helping our clients implement 

18 either custom develop or integrate custom, large, complex 

19 systems integration, and I believe I can contribute to 

20 this panel because we have extensive methodologies that 

21 have been used both within the financial services sector, 

22 within the financial market sector, and also through the 

23 many other sectors that IBM serves. 

24 So, again, thank you, and I appreciate the 

25 opportunity to participate. 

0033

1 MR. ROSS: Hello. My name is Jon Ross. I work 

2 for -- am CTO for GETCO, LLC. I've been there for about 

3 five years. Previous to that I was the CTO for the 

4 NASDAQ Stock Market, and I was the executive responsible 

5 for their single book integration of SuperMontage INET 

6 and BRUT platform. So I've got a few big software rods. 

7 My background is software development. I 

8 worked at Microsoft, did some videogames in the past. 

9 GETCO is a market maker. We primarily make 

10 two-sided markets in the public market. That's our 

11 primary strategy. We also run a customer service 

12 business and our footprint in the market is fairly large. 

13 You know, our view is the markets are going a 

14 sort of multi-decade transformation from a largely manual 

15 market to a largely automated market. They're also going 

16 through a transformation in complexity, but this isn't 

17 happening in a vacuum. The SEC has periodically visited 

18 the growing technology of the market, and most notably 

19 the implementation of ARP in '89. That was clearly wise 

20 given the events and what happened after that; expanding 

21 ARP to ATSes in '98. So I think it's highly appropriate 

22 that the SEC is visiting this once again and hopefully 

23 will shepherd us into a more stable system. 

24 I want to echo some things people said. As our 

25 markets get more complex, we're also driving a great deal 

0034

1 of value out of them, but as the complexity goes up, 

2 there's risk involved. Anyone who drives a car or uses a 

3 computer understands that cars get more complex. 

4 Computers, the Internet, all gets more complex. They 

5 tend to be prone to errors and disruptions. 
6 Minimizing that is critical, and then as the 

7 Commission realizes that, and also when it does happen, 

8 dealing with it in the most appropriate manner and 

9 dealing with the built in conflicts in management crisis 

10 I think is a very important topic as well. 

11 Thank you. 

12 MR. R. COOK: Thank you. 

13 And, again, thanks to all of you for agreeing 

14 to be here today. 

15 COMMISSIONER GALLAGHER: Hey, Robert, can I 

16 interrupt you for one second? 

17 MR. R. COOK: Yeah. 

18 COMMISSIONER GALLAGHER: In your introductions 

19 earlier you mentioned the Commission. I was on the phone 

20 before I came down with Commissioner Aguilar, who has 

21 been very ill and barely has a voice, but wanted me to 

22 tell everybody that he's watching on the Internet the 

23 Webcast. So big brother is watching, and my guess is 

24 that he'll send us an E-mail if he has any questions, but 

25 I just wanted to point that out to everybody. 

0035

1 MR. R. COOK: Thanks. I appreciate that, 

2 Commissioner Gallagher, and sorry to hear that he's not 

3 feeling well today. 

4 So again, we appreciate your willingness to 

5 spend the time with us today and to share your 

6 perspectives on these issues. 

7 I just want to note for the audience here and 

8 those who are watching that most and perhaps all of our 

9 panelists have actually submitted written comments to our 

10 public comment file with their observations and 

11 recommendations to the Commission, and these are 

12 available for anyone to review on the Commission's 

13 Website, and I want to take this opportunity to note that 

14 the comment file for this roundtable will remain open, 

15 and we invite any interested parties to add their 

16 thoughts and observations which you can do through the 

17 Website. 

18 We have already received a number of very 

19 thoughtful comments and recommendations in this way from 

20 a variety of different perspectives, and our staff will 

21 be reviewing all the comments we receive in the file. 

22 So with that, let me ask Dr. Leveson if you 

23 would like to begin with your sort of stage setting 

24 remarks, and we'll have to pass the mic down. 

25 DR. LEVESON: Thank you. 

0036

1 First of all, thank you for inviting me here 

2 today to share my experience. I apologize in advance 

3 that I'm going to have to disappear early. I have sort 

4 of triply booked today. 

5 First, let me give some additional background 

6 that wasn't clear from the official bio, and that's that 

7 I've been in software engineering now for 47 years. I 
8 first worked as a system engineer for IBM, and then I 

9 went back to school for a Ph.D. in computer science, and 

10 since then I've been teaching and doing research in 

11 software engineering. 

12 I've also been a co-owner of a software company 

13 for 20 years. So I have a lot of experience, and I've 

14 worked basically in every industry in the world. I was 

15 originally in the computer science department, a 

16 professor for the first 18 years of my career, and then 

17 moved to aerospace engineering for the last 14 for a 

18 variety of reasons that are irrelevant today, but I am 

19 not just an aerospace engineer. In fact some aerospace 

20 engineers say I'm just a computer scientist in disguise. 

21 So let me tell you a little bit of what I've 

22 learned in the last 47 years. The first lesson is that 

23 all software contains errors. I have not in all of that 

24 time ever come across any software that was not trivial 

25 in which no errors were found during operations. The 

0037

1 errors may not surface for a long time, but they're 

2 lurking there and waiting until just the right conditions 

3 occur. 

4 There are also some myths about certain 

5 industries being able to create perfect software but 

6 unfortunately this is patently untrue. No industry 

7 creates perfect software. So let me tell you some 

8 examples of things that we've -- you may have heard. 

9 So, for example, the space shuttle was rumored 

10 to have zero defect software. I always chuckle when it's 

11 always called zero defect software. When I chaired a 

12 committee for NASA, they examined the software processes 

13 for the shuttle software in 1992. We had already by that 

14 time found 27 safety critical errors in the shuttle 

15 software that could have brought the shuttle down under 

16 certain different circumstances, and they were sort of 

17 lucky. None of these, of course, led to a disaster, but 

18 at that time, just so this may be of interest, 1992, NASA 

19 was spending $100 million a year to maintain that 

20 software. 

21 Now, most places, even government agencies 

22 can't spend $100 million a year just on maintenance of 

23 software, and that was relatively simple software. It 

24 was only about 200,000 lines of code, whereas jet planes 

25 now, commercial planes have about five million. The 

0038

1 military aircraft have 15 to 20 million. Cars have 

2 upwards of over 50 million lines of code in them today. 

3 So we're talking about simple software, and 

4 they were spending as I say to get such good results 

5 from their software $100 million a year; probably spent 

6 more in later years. 

7 What about newer spacecraft? That was an old 

8 one. Well, as the use of software has grown in space so 

9 have the problems. NASA pours an enormous amount of 
10 money into software and employs the best engineers in the 

11 world. They know that their projects take upwards of a 

12 decade, ten years to create, and they get one chance and 

13 that's it. If they fail, it's all over. 

14 For example, every -- oh, and with all this 

15 effort and expense, they still have lots of errors and 

16 losses because of software. For example, every Mars 

17 mission has had serious software problems, and some of 

18 them occurred when the software could be put to sleep. 

19 What they do is essentially put it to sleep until the 

20 NASA JPL engineers on the ground can figure out how to 

21 fix it and what to do about it, and then they send up a 

22 patch to the software. 

23 In other cases, for example, the Mars Polar 

24 Lander, the Mars Climate orbiter and others, it wasn't in 

25 a situation where they could put it to sleep and they 

0039

1 just lost everything. 

2 So what about aircraft? All right. We all fly 

3 on aircraft. I don't mean to make you scared. We do a 

4 pretty good job with aircraft, as you know. We have a 

5 wonderful safety record, and most accidents that do occur 

6 are blamed on pilots because it's easy to blame pilots. 

7 There's a lot of reasons why, but the fact is that most 

8 of these accidents blamed on pilots at the very least, 

9 the software helped induce the pilot error, and sometimes 

10 the software more directly created the problems. 

11 We're probably been reading about the recent 

12 Air France 447 loss when it was flying from Rio to Paris, 

13 and that is, again, blamed on the pilots, pilot training. 

14 As it turned out it was really a technology and a 

15 software problem down underneath. 

16 And remember there are tremendous efforts to 

17 make this software failsafe and fault tolerant. Air 

18 traffic control centers have had outages for hours and 

19 had to shut down operations at airports and just tell the 

20 pilots to go wherever they can and land because they have 

21 unexpected problems. 

22 I could go on and on and with other industries. 

23 The stories are all the same, and all of these losses, 

24 the software was constructed by highly intelligent, 

25 highly skilled people who thought that they had adequate 

0040

1 backups and that they had prepared for every contingency. 

2 The very highest software engineering and technology 

3 standards were used. They did everything that we know 

4 how to do or tried to do. 

5 And things are unfortunately getting worse, not 

6 better, and why are they getting worse? Well, because we 

7 keep wanting, the way humans are want to do, we keep 

8 wanting to make things more complex, get more 

9 functionality, do more things with computers. They're 

10 almost considered magic devices. 

11 And one of the things about computers and on 
12 software is we have pure design, design that's abstracted 

13 away from the physical realization of the device. So we 

14 can build things that are much more complex because we 

15 don't have all that physical constraints that we had to 

16 deal with, and so what we're finding is that we can't 

17 anticipate all the potential unsafe and dysfunctional 

18 interactions among the components. It's not necessarily 

19 just individual component failure. In a lot of these 

20 accidents each individual component worked exactly the 

21 way it was expected to work. It surprised everyone in 

22 the interactions among the components. 

23 At the same time our attempts to create 

24 failsafe or fault tolerant systems that already use human 

25 monitors or back-ups haven't been terribly successful for 

0041

1 a variety of technical and psychological reasons. I 

2 don't have time to go into them, but basically it's 

3 almost impossible for humans to monitor computers. 

4 One of the reasons is that they're too 

5 reliable. If they failed more often, it would be much 

6 easier to monitor them, but if something only makes a 

7 mistake every six months or every year, how do you keep 

8 alert? How do you maintain the ability to catch errors? 

9 And after a while there's this thing called the 

10 incredulity response. We just don't believe that the 

11 computer, since it always does the right thing, that it 

12 could possibly be doing something wrong, and so people 

13 are loath to intervene. I mean, I'm not sure what 

14 happened in Knight trading, and it would be fascinating 

15 to talk to the people, but I have a feeling that some of 

16 these well known principals were probably involved there. 

17 Kill switches, panic buttons, we put those in 

18 all of our systems. Unfortunately they don't always 

19 work, and sometimes, in fact, our protection features are 

20 the actual things that bring us down. The problems are 

21 in our attempts to provide extra protection. 

22 And I haven't even mentioned the problems of 

23 security and software, which you're all aware of because 

24 we all deal with those all the time. So I'm not even 

25 going to talk about that, but you know it's almost 

0042

1 impossible to build. Almost? It is impossible to build 

2 totally secure software systems. The Pentagon has been 

3 trying for a long time, and they have the best minds on 

4 the planet working, and there are still people breaking 

5 into their systems. They don't like to talk about it, 

6 but it's happening. 

7 But some groups have been more successful than 

8 others. So we can learn from the groups that have been 

9 successful and that have tried very hard to do things 

10 right. Of course they use the best software engineering 

11 principles. They clearly think of quality, use high 

12 quality assurance methods, testing, others. I mean, they 

13 just do the ultimate of what we can possibly do, and they 
14 take sometimes a decade doing this before, for example, 

15 in the spacecraft software. 

16 So while I'm not suggesting that anyone 

17 shouldn't use the highest standards, it's not going to be 

18 enough. I wish it were. It's not. So how do the most 

19 successful industries limit risk? They use three 

20 practices in addition to the standard technological 

21 practices. 

22 One of these is oversight. Most industries 

23 that have very high reliability software have government 

24 agencies overseeing what's being done with an iron hand. 

25 For example, the FAA and the NRC, Nuclear Regulatory 

0043

1 Commission, the aircraft industry knows that people will 

2 stop flying and we finally would have high speed rail in 

3 this country if the planes start falling out of the sky, 

4 and if they don't exercise the utmost discipline in 

5 creating and maintaining aircraft and air traffic control 

6 software. 

7 So they participate. They understand that 

8 public confidence is a critical part of their industry, 

9 and they do as much as the FAA. The FAA is really 

10 working with them in partnership. It's surprising, and 

11 the industry itself writes its standards and does a very 

12 good job of it because they want to stay in business, 

13 frankly. 

14 The nuclear power community until recently just 

15 didn't allow software in the nuclear power plants. 

16 Unfortunately, we're now starting to build fully digital 

17 nuclear power plant protection systems, and the Nuclear 

18 Regulatory Commission is starting to provide strict 

19 standards and oversight. One of the meetings I'm 

20 supposed to be at right now -- I sent a grad student 

21 instead -- is with the Nuclear Regulatory Commission to 

22 help them figure out what they should be doing. It's one 

23 of my best graduate students though. 

24 MR. R. COOK: Yeah, feel free to leave. 

25 DR. LEVESON: I promise. 

0044

1 (Laughter.) 

2 DR. LEVESON: One of the wonderful parts about 

3 being at MIT is your grad students are all brighter than 

4 you are. 

5 Other agencies haven't been so on top of 

6 things, and they all are starting to be concerned, such 

7 as the FDA, which is the other place I'm supposed to be 

8 right now. You would be appalled if you knew how many 

9 people are killed by software and medical devices every 

10 year. I mean no one wants to talk about it, to be 

11 honest, because the numbers are large. 

12 The first best practice is that the government 

13 oversight has been used in these industries. First of 

14 all, they understand, as in your industry, that public 

15 confidence is critical for the survival of the industry, 
16 and the government and the industry have worked hand in 

17 hand to ensure that public confidence is not disturbed. 

18 The second practice they use is essentially 

19 being extremely conservative in the technological 

20 devices. Now, it's not that they don't use the latest 

21 technology. They do. They use the latest and greatest 

22 technology, but they limit software functionality and 

23 complexity. So the software in these successful 

24 industries contains for the most part only the minimum 

25 function that's required to achieve the goals of the 

0045

1 system. It doesn't add on stuff. You don't mix up the 

2 entertainment software on your airplane with the other 

3 software. 

4 And unlike the financial industry with its high 

5 frequency trading, the other industries that require 

6 public confidence to survive limit the complexity of the 

7 software they build and they only implement, as I say, 

8 the basic functionality they need to get away with to 

9 serve the primary mission. Any extra stuff, 

10 entertainment systems and stuff is strictly separated. 

11 People want to do that; you want to have 

12 entertainment on planes, absolutely important, but you 

13 don't mix that up with your avionics software. 

14 The third and final practice I want to talk 

15 about is the application of systems thinking and system 

16 engineering. These industries realize the problem is not 

17 just a technology problem; that they need to design the 

18 larger system so that software errors don't cause mayhem 

19 because they know that the software errors are going to occur 

20 despite what they do. 

21 And they do this through providing the control 

22 structure that limits and controls risk by enforcing 

23 constraints on the non-technology related system behavior 

24 that preclude or at least greatly reduce serious losses. 

25 They understand that they need to fix the system, not 

0046

1 just fix the technology. 

2 I wrote about how to do this in my new book. 

3 I'll get a little plug: Engineering: A Safer World, 

4 which was published last January and is getting a lot of 

5 attention, and it's really a different approach, but it 

6 talks about how do you apply systems thinking. How do 

7 you look at the larger system and fix that, too, along 

8 with the technology? 

9 So sort of to summarize, I don't want to sound 

10 like Chicken Little or a latter day Luddite. I did not 

11 get a Ph.D. in computer science and spend 47 years 

12 working in the field just to try and convince everyone 

13 not to use computers. But the bottom line is that 

14 there's 100 percent certainty that you will have more 

15 upsets caused by the financial system software and 

16 probably in not too long a time, but it will occur, and 

17 it's probably going to start occurring unless something 
18 is done more frequently because people are going to keep 

19 adding more functionality and more risk into the system 

20 unless it's limited. 

21 And there is no technical fix. It doesn't mean 

22 we shouldn't test, as people were talking about. Use the 

23 highest quality assurance methods, do everything we can 

24 to build great software, but that's not going to totally 

25 solve the problem. 

0047

1 The industries that have learned this lesson 

2 the hard way limit their risk with discipline and 

3 establishing controls. The biggest mistake that the 

4 Titanic designers made was believing that they could 

5 build an unsinkable ship and, therefore, they didn't have 

6 to prepare for contingencies, for calamity. 

7 And this is still true. We've learned better 

8 in these very safety critical industries. We've learned 

9 that we cannot build an unsinkable ship and we cannot 

10 build unfailable software, perfect software. The 

11 financial industry needs to learn, too, that computers 

12 aren't magic; that our engineering techniques for 

13 creating software aren't perfect; and that failsafe and 

14 fault tolerant designs, whether these features are 

15 automated or they use humans in a monitoring function, 

16 are a goal but not yet a reality. 

17 We need broad approaches and solutions that go 

18 beyond the technology. If instead this industry engages 

19 in hubris and wishful thinking, we're all going to have 

20 to live with the consequences. 

21 Thank you. 

22 MR. R. COOK: Thank you, Dr. Leveson. We 

23 really appreciate your insights, and I think they provide 

24 a valuable backdrop and context for the rest of our 

25 discussion. 

0048

1 I know you need to leave. So we appreciate 

2 your spending your time here. I suppose there are 

3 probably a number of questions those of us who aren't 

4 flying today might like to ask, but in the interest 

5 of making sure that we have time to hear from our 

6 other panelists, I'm going to move us on to that unless 

7 there's any objection, if anyone has any. 

8 Again, thank you for joining us today. 

9 DR. LEVESON: Thank you. 

10 PREVENTING ERRORS THROUGH ROBUST SYSTEM DESIGN, 

11 DEPLOYMENT, AND OPERATION 

12 MR. R. COOK: In some respects I guess, you 

13 know, your comments cast new light on the title of this 

14 panel, "Preventing Errors," but we're eternally 

15 optimistic, and so we will begin to explore this through 

16 a series of questions that we have for our panelists. 

17 And the first one is really about industry best 

18 practices, and maybe I'll ask Christopher to take the 

19 first crack at addressing this, given your kind of 
20 perspective working with a variety of different financial 

21 services firms. 

22 But can you talk to us a little bit about what 

23 are the industry best practices for testing robustness, 

24 deployment, and the use of software systems? In 

25 particular, are they sufficient today to support the 

0049

1 continuity and integrity of the markets? And if not, 

2 what more from your perspective needs to be done? 

3 MR. RIGG: Sure. Well, first I would say that 

4 I think when you talk about best practices, I think best 

5 practices are aligned to particular industries, and 

6 different industries have different levels of risk that 

7 they face in the marketplace, and that tends to drive the 

8 practices that they choose. 

9 So just as Dr. Leveson talked about when the 

10 risk of what you're deploying into production goes up, 

11 the amount of rigor typically that you apply to that 

12 process should go up commensurate with that. So higher 

13 risk applications should have more process rigor 

14 associated with them. 

15 So I think from a best practice perspective 

16 we're seeing that through many industries where you can 

17 see in the financial services industry you have very 

18 large, complicated systems that have to be up all the 

19 time. If you look at the Fed. wire system, if you at ATM 

20 systems at large banks, those systems have a great deal 

21 of rigor associated with them, which means very small 

22 number of changes, an extensive amount of testing prior 

23 to those changes going into production, and a significant 

24 amount of sign off by the various stakeholders. 

25 So I think, you know, it's hard to apply a 

0050

1 single practice to any particular industry because 

2 obviously some industries have a great need to have a 

3 high frequency of change because they feel like they need 

4 that in the marketplace in order to respond to the 

5 competitive nature, and I think this industry is 

6 certainly one of them where there's a significant amount 

7 of change. So the tendency can be to use less rigor when 

8 there's more frequency of change. 

9 I would say in the best practices space you 

10 need to continue to figure out a way to balance those 

11 two, the amount of rigor with the frequency of the need 

12 to make change very quickly. 

13 In terms of whether they are adequate, I think, 

14 you know, I'm not sure I'm in a position to determine 

15 that. I think that every business needs to look at their 

16 processes, the key stakeholders. One of the best 

17 practices, I think that we all who develop software know 

18 that you need to make sure that the key stakeholders, 

19 which typically includes the business, whoever owns the 

20 P&L, they need to be integrally involved in the process 

21 so they understand what they're asking for, the risk 
22 associated with what they're asking for, sign off and 

23 agree to the amount of testing that's going to be applied 

24 because ultimately at the end of the day all of us here 

25 who are technology people, we typically don't run our 

0051

1 businesses. We are providing the technology that the 

2 people who own the businesses are providing. 

3 So in that case I think one of the key things 

4 there is to make sure that the business understands the 

5 risk associated with the technology being developed and 

6 they agree to the level of rigor that's being applied. 

7 CHAIRMAN SCHAPIRO: Robert, could I jump in 

8 with a question? 

9 I thought that was a great presentation from 

10 Dr. Leveson, and one of the things that really struck me 

11 was this idea that in the Nuclear Regulatory Commission 

12 world or the FAA, they're very conservative about their 

13 use of technology, and they go towards, you know, just 

14 dealing with the essential functionality and not lots of 

15 add-ons. 

16 And I wondered if in your experience at IBM and 

17 really for anybody in the financial services industry has 

18 that discipline of let's focus on exactly what must 

19 happen with this software and not throw in lots of bells 

20 and whistles, which as she describes it leads to the 

21 potential for the interaction of components that creates 

22 problems. 

23 MR. RIGG: I'll start and encourage other 

24 people to contribute. 

25 I would say that I think in most cases for key 

0052

1 mission critical applications that have certain 

2 requirements associated with them, whether it be high 

3 speed of execution, obviously which is something that's 

4 common here, I think in those cases I think most of the 

5 time the software is minimized to what it actually needs 

6 to be, and often that is simply because of the 

7 requirement to execute at the speed that they need to in 

8 these marketplaces. 

9 So I think if you were to look at the core 

10 routing algorithms and core processing software of any of 

11 the people up here and any of the participants in the 

12 industry you would find that that software is fairly 

13 lightweight, fairly focused specifically on the task and 

14 doesn't have a lot of extraneous feature and 

15 functionality. 

16 I think there are other parts of, you know, the 

17 organization of many businesses that have software that's 

18 more feature rich and more focused on that because they 

19 feel like they need that in the marketplace, but I think 

20 for the key areas of financial systems that are 

21 performing very specific tasks, I think that that 

22 minimization principal is generally applied, and what 

23 I've seen is it applied pretty well and pretty 
24 appropriately. 

25 MR. ROSS: I have a quick comment. The most 

0053

1 stable platform I ever worked on, which was also the 

2 simplest, and I don't think that's coincidence, which was 

3 Ireland, just to bids and asks, matched them. No 

4 opening crosses, no pegged orders, nothing fancy at all. 

5 This was obviously pre- Reg. NMS. That was by far the 

6 most stable platform I ever worked on. 

7 That platform we decided is not appropriate for 

8 today, you know, with all of the interlinkages between 

9 exchanges, open and closing processes that need to be 

10 implemented, but I think it's maybe worth -- I think it's 

11 maybe worth reflecting on for a minute how much of the 

12 complexity in the system do we actually need, to your 

13 point exactly. 

14 MR. LAUER: I'd like to say that I think 

15 sometimes from what I've seen we can go too far actually, 

16 especially in low latency systems, when we focus simply 

17 on what the task is for that software to perform. I've 

18 seen risk checking being neglected in favor of faster 

19 performance, that type of thing. 

20 So on one hand, you can go too far in 

21 simplifying too much, especially when there is this 

22 latency race to zero. On the other hand, it can be very 

23 tempting by designers of these systems who are generally 

24 very smart to see them as much simpler systems than they 

25 actually are, and so when they go to make changes to what 

0054

1 they perceive as a simple system, they don't always 

2 understand the complexity of even minor and small 

3 changes, and especially the complexity of how these 

4 algorithms are interacting with one another in the 

5 marketplace, and that can lead to much less rigor in 

6 terms of testing systems both from a quality assurance 

7 perspective where you tend not to have independent 

8 quality assurance groups in many firms, as well as a load 

9 testing perspective where software is not adequately 

10 exercised in conditions that, you know, accurately mimic 

11 what the market conditions are like when you're actually 

12 trading. 

13 MR. ARYA: I'd like to add to your point. I 

14 think isolating smaller components is extremely 

15 important, but I think also taking, to kind of complement 

16 that, taking the approach to have a multi-shell approach 

17 so that you will have smaller components that are written 

18 with simplicity, but in order for risk checks and so 

19 forth, we take a very multi-layered approach. 

20 So you have other risk systems they are 

21 continuously observing how these low latent systems are 

22 trading, and they have means to intervene in real time. 

23 And it's extremely important to keep these low latency 

24 systems very, very simple, but also it's very important 

25 to have integration between these outer shells that 
0055

1 protect these simpler systems and continuous integration 

2 testing between the two. 

3 We run a dark pool, one of the largest dark 

4 pools, and we specifically go out of our way to make sure 

5 that the actual crossing engine itself is extremely 

6 simple, but that said, its interaction with tape 

7 recording, its interaction with order writing and so 

8 forth has to be continuously tested. The crossing engine 

9 could be all doing well, but if the external components 

10 are not doing well and interacting properly, that could 

11 lead to a disaster. 

12 So having smaller pieces as simple as possible 

13 but having outer shells that protect them, watch them is 

14 equally important. 

15 MR. NAZARALI: If I could just make a point, 

16 we're spending a lot of time talking about software 

17 development and testing and implementation, and that's 

18 very important and all of us can do a better job doing 

19 that. But as we think about trying to avoid the type of 

20 massive errors that happened at Knight Capital Group, 

21 it's important to think about the whole system. 

22 And as Dr. Leveson pointed out, two things 

23 really resonated. Number one is all software contains 

24 errors, and number two, it's not just a technology 

25 problem. 

0056

1 So if you look at what happened at Knight 

2 Capital on August 1, the first five minutes of that 

3 trading was really a software problem. The next 35 

4 minutes where the software was not shut off was really a 

5 risk management and control and management processes 

6 problem. And I think it's important for all of us as we 

7 make our systems more robust and improve how we implement 

8 the software, that we also put in place the right 

9 management and risk protocols so that if something like 

10 that happens we can pick up the phone and call the New 

11 York Stock Exchange and say, "Shut off all trading. Kill 

12 all open orders." 

13 If that had been done five minutes after on 

14 August 1 at Knight Capital, we probably wouldn't be here 

15 right now. So I think that that's something that's very 

16 important for us to all consider. 

17 MR. PASTINA: Just another point that Dr. 

18 Leveson mentioned was that the amount or number of lines 

19 of code in a car, and that really struck me because she 

20 was talking about millions of lines of code in a car, and 

21 I don't know about the other participants, but a typical 

22 matching engine doesn't have millions of lines of code. 

23 It typically has thousands of lines of code. It's much 

24 smaller. It's much more streamlined. It's not anything 

25 like that, and you know, comparing the two is very 

0057

1 interesting because the amount of testing that goes into 
2 a car compared to the amount of testing that goes into a 

3 matching engine, it's a much smaller component. It's not 

4 as big as, you know, a couple million lines of code. 

5 MR. ARYA: I think in terms of best practices, 

6 I would like to comment that it's really -- Dr. Leveson 

7 said that errors are inevitable. I fully agree with 

8 that, but I think accepting that and understanding that 

9 errors are inevitable and actually really cultivating a 

10 prevention culture in your team is extremely important. 

11 Before the first line of code is ever written, 

12 what design principles go into play? To your point, we 

13 think that if technologists are really interacting with 

14 the business folks, folks that manage P&L, and they say, 

15 "Okay. I'm writing a line of code where I'm about to see 

16 an erroneous trader, an erroneous code. What do I do?" 

17 and if they're not integrated well with business, their 

18 temptation or instinct might be, "Well, I'll throw an 

19 alert and good things will happen." 

20 But if the two teams are very well integrated, 

21 they will go and talk to the product manager. They will 

22 go and talk to the business guys and say, "What's next?" 

23 and that really should go into our design phase, talking 

24 about what error conditions happen. 

25 Everybody talked about errors, and they will 

0058

1 happen. What really gets us, to Jamil's point, is double 

2 errors. Software errors happen, and there's a cascading 

3 effect, and that really needs to be integrated into our 

4 software design. 

5 Error prevention at first level is very 

6 possible, mostly doable, but it's the double failures 

7 that are not planned for, and we try to induce that 

8 culture where you really talk about there was an exchange 

9 issue or a broker issue on the other side. Why do some 

10 of the networking errors happen? Or how do you prevent? 

11 You try to pull off a kill switch that is 

12 internal for a client, and it didn't work. So double 

13 failures really need to be built into these best 

14 practices and need to be thought of when you design the 

15 system before you write the code ever.

16 MR. R. COOK: I think that's an important point 

17 and Jamil made the point earlier about it's not just the 

18 front end. The back end is sort of in a way contesting 

19 the logic of the two panels we have because they do 

20 overlap in certain respects, and I think we recognize 

21 that. 

22 But coming back to the best practices question 

23 for a minute, maybe you could give us a little inside 

24 window into is that even the right term in this industry. 

25 Are there commonly understood best practices? 

0059

1 A lot of you have worked in different 

2 organizations. As you move from one to another, does the 

3 best practice of that organization for developing code, 
4 for testing it, for rolling it out, does it look similar 

5 to where you were before? 

6 When you talk horizontally across different 

7 types of market participants, is the idea of best 

8 practice sort of -- does it exist regardless of the 

9 nature of the entity you're working for? Does it exist 

10 at all? 

11 And I'll open that up to anyone on the panel. 

12 MR. PASTINA: If I can jump in just quickly --

13 MR. R. COOK: Sure. 

14 MR. PASTINA: -- I mean there is a system 

15 development life cycle. There's an idea. That idea has 

16 to be documented typically in terms of specifications. 

17 Those specifications get turned into, from a business set 

18 of specifications, into a set of specifications that a 

19 coder can understand, so a design set of specifications. 

20 There's a coding phase, and after the coding 

21 phase there's a testing phase and implementation phase. 

22 So that cycle exists everywhere. However, how that cycle 

23 is implemented is widely different. So in a more 

24 entrepreneurial type environment, the specification is 

25 written on a napkin, right? Someone has an idea in a bar 

0060

1 somewhere and they say, "This is a great idea. Here's 

2 the idea. Here's the spec coded." 

3 And it goes right into coding. Somebody looks 

4 at it and says, "Yeah, this is about what I expected to 

5 see coming out of coding," versus a more disciplined 

6 approach where an idea has to pass through a set of 

7 filters. It then becomes a specification, and the least 

8 expensive place to weed out errors is early on in the 

9 program. So the most expensive is obviously in 

10 production, testing, coding, but the least expensive is 

11 in the specification phase. 

12 And so there are techniques for inspecting 

13 specifications in a very disciplined way to weed out 

14 errors up front where it costs you very little to do. 

15 MR. NAZARALI: If I could just make a point, we 

16 do believe that there are software best practices, and 

17 they really fall into a couple different categories. One 

18 is software best practices just in the development 

19 implementation of software, so things like, you know, 

20 unit testing and regression tests and things like that. 

21 The second set of software best practices 

22 really relate to very specific firms. So, for example, 

23 our software development practices in an automated market 

24 making business are going to be very different than the 

25 software development best practices for an ATS or for an 

0061

1 agency trader. 

2 One of the really powerful things that came out 

3 of what happened on August 1st is that Wall Street firms 

4 across the industry said to their technology teams, "We 

5 want to make sure this never happens to us. So go back 
6 and, you know, make sure that the system integrity and 

7 how you roll things out and the checks are such that the 

8 probability of this happening is miniscule." 

9 And as an industry, we met with other firms. 

10 We met with both competitors and customers, and we shared 

11 some of the best practices, and many of them were things 

12 that we were already doing and things that they were 

13 already doing, but there were some ideas there that both 

14 we and some of the firms that we met with found very, 

15 very helpful. 

16 For example, at Citadel, we have developed this 

17 fuse box technology over the last ten years, and it's a 

18 system that sits outside of your trading software, and it 

19 listens to all the trades and executions that happen, and 

20 under certain conditions much like your fuse at home will 

21 trip, if the ADV that you're trading is much higher than 

22 a set parameter, if your risk limits are much higher, if 

23 your P&L goes out of bounds, it will trip and then a 

24 human will then have to turn it back on. 

25 So things like that were not necessarily things 

0062

1 that other firms were using, and that was very helpful as 

2 we shared best practices. 

3 MR. ISAACSON: I'd like to also mention as far 

4 as best practices and as I mentioned in my opening 

5 remarks, I think everyone here is probably doing unit 

6 testing of the different components, but something that 

7 Dr. Leveson mentioned was that usually where the errors 

8 arise are from the interactions between components that 

9 are operating perfectly. So integration testing and 

10 regression testing is paramount. 

11 And you can do that within a firm, and it's 

12 probably very similar within firms whether you're an ATS 

13 or an exchange or a market making firm. However, what's 

14 very difficult is to simulate the market. How does the 

15 market actually interact with 13 exchanges and many dark 

16 pools? 

17 And that's why I'm advocating that all market 

18 centers support test symbols, and then each strategy is 

19 rolled out using a test symbol in production environments 

20 where firms like Jamil's are interacting. You know, 

21 they're testing their new strategy in real live 

22 production environments where real live quotes are being 

23 taken, but using a test security that doesn't clear, 

24 that's not printed on the tape, but everyone can see all 

25 of the interactions, especially, you know, the designer 

0063

1 of the system. 

2 And the exchanges can do this as well with new 

3 order types. That's really the truest test. That's the 

4 final test once you've gone through unit tests, 

5 integration testing, and then user acceptance testing. 

6 This would be the final test where you're testing with 

7 test symbols in a production environment. 
8 Now, realizing that errors will occur and we 

9 need to have things on the back end to make sure that 

10 they're, you know, remediated immediately and stopped, 

11 but I think that would go a long ways in our environment. 

12 COMMISSIONER GALLAGHER: Can I jump in on that 

13 real fast? 

14 Chris, I guess the question is you just can 

15 never replicate the real market for testing purposes. 

16 You can take, you know, a reel from yesterday's trading 

17 and test into that and expect that it's as good or 

18 vibrant as --

19 MR. ISAACSON: I'm not going to say never. I'm 

20 saying it's an incredibly difficult thing to take in all 

21 of the market data feeds and in perfect sequence 

22 regenerate them back and consider how they would interact 

23 with your matching engine or your algorithm. 

24 But you know, that's why you need test symbols 

25 to test that new strategy in the real live environment. 

0064

1 MR. LAUER: I'd like to say that that doesn't 

2 mean that you don't try to mimic what a real market looks 

3 like, and in my written comments I talk a lot about this, 

4 which is the difference between software replay and 

5 hardware, which can do what's called temporarily accurate 

6 replay. 

7 Market data comes in in the modern market very 

8 fast and is subject to conditions that are called 

9 microbursting, and that is when multiple servers can 

10 write to the line at once resulting in temporary network 

11 saturation, and that can lead, when you have complex 

12 multi-threaded software, that can sometimes lead to 

13 nondeterministic behavior with thread contention. 

14 A very technical conversation, but the main 

15 point is that many firms do not use hardware replay for 

16 temporarily accurate replay, and the ones that do, maybe 

17 they do so initially when the strategy is designed, but 

18 not as it is changed over time. For most firms -- and 

19 this is especially according to the Chicago Fed. report -

20 - you know, they found out that a lot of firms will make 

21 small changes and constantly push stuff out, or if they 

22 come up with a new strategy, they'll back-test it and 

23 say, "Hey, this is going to make some money," and put it 

24 into production very quickly. 

25 In terms of a software development life cycle, 

0065

1 I do believe there are best practices that can be adhered 

2 to, and exactly as was being described, its documentation 

3 specification is the step that is normally not taken at 

4 least for many firms. 

5 There are certainly mature firms -- Citadel and 

6 GETCO are great examples -- of very mature firms with 

7 very mature processes, but we live in a world in which a 

8 single server can send out 100,000 orders per second, and 

9 so we live in a world in which small start-ups and small 
10 firms that want to gain an edge and want to move fast 

11 will definitely lean towards not adhering to a software 

12 development life cycle and not properly testing and 

13 dealing with the types of things that you asked about in 

14 the preparation, which was the behavior and the 

15 unexpected condition testing. 

16 And I think that there are things that can be 

17 done in terms of identifying certain times, certain 

18 market events that should be mandatory testing perhaps or 

19 saying that if you do have low latency systems, they do 

20 need to be testing with hardware and temporarily accurate 

21 replay data, which is a major step. 

22 And then I completely agree that life testing 

23 in a fragmented market with test symbol ranges is a 

24 fantastic idea as well. 

25 MR. R. COOK: I think Commissioner Paredes had 

0066

1 a question, and then we can continue. 

2 COMMISSIONER PAREDES: Yes, and it really feeds 

3 into the discussion so that it may inform what your 

4 answers are going to be, those of you chiming in, and 

5 that keeps us on the theme of best practices. And a lot 

6 of general categories of issues have been identified, 

7 documentation, testing and all the rest, but as we try to 

8 get more granular, how do we know whether or not 

9 something is a best practice or not? 

10 And if you think about testing, how do you know 

11 whether or not there's been enough testing? I'm sure you 

12 can always anticipate. Well, we can do one more round of 

13 testing, one more round of testing. We can always do a 

14 little bit more. 

15 So in part it strikes me it comes back to 

16 questions of risk tolerances, as well, and tradeoffs and 

17 the like, but identifying and finding agreement that 

18 there should be best practices or there should be testing 

19 and there should be testing of this type versus that 

20 type, how do you all go about evaluating it? 

21 As a business matter, it strikes me it's 

22 important for policy makers, too, to have the benefit of 

23 that kind of more granular insight to figure out, all 

24 right, at some point a decision needs to be made as a 

25 business matter, as a regulatory matter that to do yet 

0067

1 another test, to implement yet another practice, to go 

2 yet another step isn't sensible given the set of 

3 tradeoffs. 

4 So from a business perspective, how do you 

5 know; how do you think about enough is enough; now it's 

6 time to go ahead and roll it out? 

7 MR. NAZARALI: I think you never really know if 

8 the testing is enough, and I think it's really just a 

9 business judgment where you look at the cost benefit of 

10 the potential error. So, for example, Citadel Execution 

11 Services is a large retail equity market maker and we 
12 roll out software changes all the time partly in response 

13 to regulatory requirements, partly in response to 

14 customer requests, partly in response to changes in our 

15 trading strategy, and whenever we do a rollout, there is 

16 a decision made where: okay, should we wait; should we 

17 do more, as you said, more testing, or do we implement 

18 this? 

19 And it's a cost benefit where you say: okay. 

20 There is a one percent chance of this creating a problem. 

21 How big a problem will it create? How many customers 

22 will it impact? And, you know, what's the magnitude of 

23 that? Okay. That's unacceptable. 

24 And you never know that percentage exactly 

25 because you may think it's one percent, but it could be 

0068

1 three percent or half a percent, but it's really that 

2 kind of cost benefit where you try to anticipate the 

3 potential cost of something going wrong. 

4 CHAIRMAN SCHAPIRO: And how do you think about 

5 the cost not just to your customers, but to the broader 

6 marketplace, investor confidence, and the customers of 

7 other firms that may be impacted by your judgment that 

8 we're ready to go? 

9 MR. NAZARALI: that's a great question, and for 

10 us because we are such a large player in the market, we 

11 do consider all of those things on the market making 

12 side. You know, we have roughly 25 percent market share 

13 in retail equity markets. So we realize that if there is 

14 an impairment of investor confidence, it's going to 

15 affect our business. 

16 The concern though is that smaller firms that 

17 don't have such a vested interest in insuring the 

18 integrity of the markets, they will make a very different 

19 calculus because when they're rolling something out, the 

20 cost of implementing investor confidence and hurting the 

21 overall market doesn't affect them nearly as much as it 

22 would affect someone like Citadel. 

23 MR. ARYA: I would like to add that regarding 

24 testing, as Jamil said, there's always a line you draw, 

25 and it's judgment based, what releases go out, and it 

0069

1 really needs to be, as he said, you know, talked about 

2 between technology and business folks. 

3 But within the testing itself, it's not just 

4 the testing of the software itself, but it's also the 

5 testing of how the software will recover if there are 

6 problems. 

7 So you may never ascertain that there are no 

8 errors, as Dr. Leveson said. However, it's not just 

9 testing that the software is functioning properly. We 

10 also look --when I say we're okay to go ahead with this 

11 release, we also look at what testing has been done about 

12 failures, what testing has been done with respect to 

13 diverting the software back to the previous version. 
14 So it's not about that the given version would 

15 work or not, but it's also about if there are unforeseen 

16 issues, how quickly can you divert back. 

17 To Chairman Schapiro's point, investor 

18 confidence and so forth, there will always be error, and 

19 we can make an error of judgment, but as long as you code 

20 and you test for diverting and curbing those problems as 

21 quickly as possible, and that needs to be tested, and 

22 your code switches need to be tested and your 

23 intervention needs to be tested, and if those are tested, 

24 then you have some degree of confidence that, yes, you 

25 can divert back. Errors will happen in these extreme 

0070

1 cases, and there are ways to interact with other systems 

2 and how you recover and how you quickly go back. 

3 So deployment of that software and diverting of 

4 that software is equally important from a best practices 

5 perspective, as is the testing, because there's the next 

6 phase. 

7 One last thing I want to mention. It was 

8 mentioned about exchange testing as well as back-testing. 

9 One of the areas where we have really got some traction 

10 is in operating our own dark pool and crossing engine. 

11 We had previously struggled quite a bit about rolling out 

12 a new software release for the crossing engine and having 

13 issues. 

14 So over the years we have actually built a 

15 parallel production testing environment where all the 

16 real time flow is also directed to the next version of 

17 software. It trades in parallel. It compares in 

18 parallel. It is actually subjected to the same kind of 

19 market data, and I think more and more firms, even in 

20 smaller scales and not everybody has the luxury to be a 

21 dark pool because it's an endpoint, so I totally 

22 recognize that for exchanges that problem is a lot 

23 harder. 

24 However, it's really paramount that, you know, 

25 some of us get together and devise those ways or talk 

0071

1 about how this parallel testing can be done because 

2 really the only way to subject your new software to real 

3 time market conditions is to actually subject it before 

4 you release it, and it's an interesting computer science 

5 problem, but that can also be worked out with talking to 

6 industry guys and business folks. 

7 MR. R. COOK: So I think we have questions from 

8 Craig and Tom, but I'd like to just wrap up the answers 

9 to Commissioner Paredes and Chairman Schapiro's 

10 questions. 

11 MR. ISAACSON: I just have one more thing about 

12 that all important decision where you're making a 

13 judgment call. Is the software ready to be released? 

14 It's not just the business involved with technology, but 

15 it's also Compliance Department, Regulatory Department. 
16 There's an exchange. Does this new functionality match 

17 what our rules say? 

18 And so you've got to have not just business and 

19 technology but compliance involved in order to really 

20 make the right risk-reward tradeoff. 

21 MR. LAUER: I'd also like to say I think that 

22 Chairman Schapiro's question is the crux of the matter. 

23 Does an individual firm adequately account for the impact 

24 it can have on the market and the broader implications? 

25 Which is why I believe there's a clear 

0072

1 regulatory role as well for any firm that has direct 

2 market access. That's the enter point where you can 

3 start mandating potential quality accreditation standards 

4 or allowing firms to use quality accreditation in their 

5 best ex decisions so that there can be at least in that 

6 regard a private market sort of race to develop better 

7 technology and better quality management standards. 

8 MR. ROSS: We've been talking about testing of 

9 individual components and when is it time to release a 

10 component. If I have a component that I'm changing, the 

11 component in front of it that's doing extra-regulatory 

12 checks I don't change, right? So I have a chain of 

13 components all doing checks. I only change one at a 

14 time. 

15 If I'm changing the one in front, I don't 

16 change the one in back that I know behaves properly, 

17 right? So nobody should rely on a single component 

18 having checks. 

19 To that point, one thing that no one has talked 

20 about except for Mr. Ratterman in front is the financial 

21 industry is unique in that you can actually see your 

22 effect on the market independently. So we have real time 

23 drop copies for most of the exchanges. What we do as a 

24 firm is we have separate code that implements those drop 

25 copies and separate code that reconciles what we think we 

0073

1 do in the market from our proprietary trading systems to 

2 what we're actually doing in the market. 

3 Almost no other industry has this. You can't 

4 have a robot that flies alongside a plane and when you 

5 push the stick down sees the surfaces turn. This is very 

6 unique. I think it's very important. 

7 So from a best practice point of view, I think 

8 there's a very key best practice. It's not necessarily 

9 in software development. It's in how your system 

10 operates and how your system reconciles. To be slightly 

11 pejorative, some exchanges make money off of their drop 

12 copies, which might not be the best thing. Also some 

13 exchanges drop copies are actually not real time. 

14 They're like way back in system. So your view from the 

15 downstream drop copy is just slow enough that you can't 

16 trust it. I think those would be interesting areas for 

17 the Commission to explore. 
18 But that independent view is critical, and I 

19 think any institution worth its salt spends the resources 

20 to build that independent view of what effect they're 

21 having in the market in real time. 

22 COMMISSIONER GALLAGHER: Before we leave 

23 testing, what are we doing now that impedes proper 

24 testing or, on the flip side, what could we do to 

25 facilitate a more vibrant testing atmosphere? 

0074

1 MR. ARYA: I think the point was made about 

2 live symbols. That would greatly help us as an agency 

3 broker. It would greatly help us, and I think the key 

4 point is that all exchanges should participate in that, 

5 and also we should encourage market participants because 

6 exchanges could provide functionality to test your 

7 symbols, but it's really a function of how much feeds 

8 actually go into it. So that's extremely important. 

9 And if there are ways to -- also I think the 

10 point about drop copy is paramount. We think that 

11 there's a lot that can be in the drop copy and done in 

12 the drop copy area; would love to find ways to make them 

13 real time for all exchanges; would also love to actually 

14 find ways to get aggregated alerts of some sort where I 

15 think one of the comments had that exchanges could have 

16 peak notional values and so forth and threshold goals and 

17 a communication comes back to the brokerage firm. 

18 But what would be very good is if that can 

19 actually also be somewhat automated. So if my firm is 

20 sending above and beyond 75 percent of what I typically 

21 send, whether it's ADV or peak notional value, having a 

22 feed in drop copy warning these firms in real time would 

23 be very, very good. 

24 And, again, guidelines to do that and industry 

25 working together would be extremely helpful because we 

0075

1 can actually test that and also have our own alerts in 

2 sync with what the exchanges are seeing. 

3 MR. ISAACSON: I think longer term there's 

4 another thing the SEC can do, and that's the forcing of real 

5 time reporting to the DTCC. We at BATS believe that 

6 while there may be interim solutions at the exchanges 

7 where we have kill switches based upon limits at each 

8 individual exchange, ultimately that functionality 

9 belongs to the DTCC, and in order for them to do those 

10 real time position limit monitoring, they need to have 

11 all the trades in real time. And I believe they have the 

12 vast majority of them in real time today, but not 100 

13 percent. 

14 And like drop copies from the exchanges, a firm 

15 can't get a full view, an independent view, without 100 

16 percent drop copies, and likewise DTCC can't make a 100 

17 percent calculation of the exposure position of that 

18 member. 

19 COMMISSIONER GALLAGHER: I'm not going to 
20 disagree with you, but I'd point out that I think their 

21 pricing structure has to facilitate real time. 

22 MR. ISAACSON: I would agree with you. I would 

23 agree with that. We have no interest in, you know, 

24 necessarily increasing the cost of clearing materially, 

25 but I think, you know, this is an important enough risk 

0076

1 management discussion that we should make both changes to 

2 make it happen. 

3 MR. NAZARALI: This may bleed into this 

4 afternoon's panel, but since you're asking what you could 

5 do, one of the things that we think is really critical 

6 and, as being part of the industry working group that 

7 submitted the recommendation, is we think that kill 

8 switches at the exchange are very, very important because 

9 as we've talked about here, we can all improve our 

10 processes, but software will always have errors, and our 

11 trading systems may -- things may slip through. 

12 And so the exchanges as the gateway, as the 

13 final stop where the trading happens are really in a 

14 position to be able to shut off this aberrant trading 

15 activity if they see it, and we think that's very, very 

16 important so that a problem at an individual firm doesn't 

17 become so large that it threatens the stability and 

18 integrity of the overall market. 

19 MR. LAUER: I have to say I believe that 

20 exchange kill switches are an important first step, but a 

21 very inadequate one. I think there is no way to approach 

22 the problem of kill switches without a market-wide 

23 perspective. Everyone is trading on different markets. 

24 No firm sees a market in isolation, and kill switches in 

25 isolation can be inadequate. 

0077

1 I think that in my written statement, I 

2 submitted an idea for a strategy registration system so 

3 that any piece of software that will have direct market 

4 access would need to be registered, assigned a unique ID 

5 number, and include that ID number in any quote that it 

6 sends to the market. While that ID number would not be 

7 publicly visible, it would still filter through down to 

8 the SEC. 

9 This is not quite the consolidated audit trail. 

10 It's what I would think of as an intermediary step in 

11 which the SEC could take on the role of market-wide 

12 surveillance mechanism, working with exchange 

13 surveillance groups. So in a distributed system, but 

14 still with a consolidated view, able to keep track of 

15 what individual strategies are doing with software, like 

16 Dr. Leveson was saying. You can't have humans monitoring 

17 software. You need advanced software to monitor 

18 software. You need low latency systems to monitor low 

19 latency traders. 

20 This is a distinction that is critical, and the 

21 one reason that I bring this up in consideration of the 
22 consolidated audit trail is that it is not real time, and 

23 while it has lots of features that don't need to be real 

24 time, real time surveillance is necessary, and with the 

25 strategy registration system, after having back-tested a 

0078

1 system, a firm can tell you what it expects this system 

2 to do. What are the characteristics of it? 

3 The SEC can also build software systems that 

4 monitor these strategies over time and develop heuristics 

5 to actually build what I would call dynamic adaptive kill 

6 switches that operate market wide. 

7 MR. ARYA: I would totally agree about the 

8 inadequacy of kill switches in a vacuum. So I think 

9 metrics for those kill switches need to be really worked 

10 out by the industry and finding out there is no one kill 

11 switch threshold that fits all. Some firms might want 

12 volume related; some firms might want notional value 

13 related. 

14 So I think a lot of discussion needs to take 

15 place for deciding those metrics. Ultimately I think 

16 kill switches should be put in place after that 

17 discussion has taken place. 

18 However, regarding strategy registration, I 

19 think it might or might not work, and I'm not the expert 

20 in the prop side of things; however, for agency brokerage 

21 firms which are really writing agency algos, I think a 

22 registration and actual auditing of that would be a huge 

23 cost to the industry, and I think monitoring of that 

24 would be hugely expensive. 

25 Just take into account the consolidated audit 

0079

1 trail. There's a lot to be done even in non-real time 

2 putting the data together and so forth. For the amount 

3 of work that needs to take place and strategy 

4 registration for firms that are continuously evolving and 

5 so forth, I think that's a step that will eventually 

6 potentially get there, but there's a lot to be done ahead 

7 of that, and jumping to just registrations and 

8 validations of technologies solely in the game I think is 

9 somewhat of an overkill. 

10 MR. R. COOK: I want to make sure no one is 

11 prejudiced by not having a mic nearby them. So, Lou, I 

12 think you maybe wanted to get in on this. 

13 MR. PASTINA: I did. I had almost forgotten 

14 what I was going to say, but going back to the question 

15 about what could you do to help in the testing area. 

16 It's amazing to me how many times software gets 

17 introduced and firms don't test with you. So whether we 

18 have test symbols in production or we run industry tests, 

19 it's always the same firms that come in and test, and 

20 those are the firms that generally don't have issues. 

21 And then there's a long list of firms that 

22 never show up. There's also --

23 MR. R. COOK: What's the balance? I mean, is 
24 it 80 percent show up or --

25 MR. PASTINA: Well, unfortunately, you know, 

0080

1 the 20 percent that show up represent most of the volume, 

2 which is good. However, all you need is someone to come 

3 in with a bad message and ruin the whole system, and if 

4 you don't have the proper defensive code in place, that 

5 message then can be propagated and because of the 

6 interconnectedness of our systems today, be passed on to 

7 other marketplaces. 

8 And so I'm not advocating that everyone be 

9 required to come in and test. I don't think you can wait 

10 for every customer to come in before you put a new piece 

11 of software in place, but there should be some review 

12 about how firms actually take advantage of the testing 

13 opportunities that are afforded them. 

14 MR. ROSS: I tend to agree. On the strategy 

15 registration, all of the errors we're talking about are 

16 not strategies behaving as they're supposed to. The 

17 strategies behaving -- or actually they're not even 

18 strategies. Most of them are just software behaving as 

19 it's not supposed to. It might have unintended side 

20 effects on strategies. 

21 It seems like if we could enumerate all of the 

22 ways that strategies would fail, then we would just fix 

23 them, right? The classic thing: in order to write fool24 proofed software, you need a really clever fool, right? 

25 So I'm not sure of the value of registering 

0081

1 strategies because when they behave oddly, they behave 

2 oddly in circumstances no one has ever seen or it's a 

3 knock-on effect from a straight up software bug. So it 

4 seems like minimal returns on that. 

5 MR. R. COOK: Let's get a -- oh, sorry. Did 

6 you have another point? 

7 MR. ROSS: Yeah, I just wanted to say, yeah, I 

8 think we need to talk a little bit like what classifies 

9 as an error, right? Is a firm not being able to connect 

10 to a marketplace an error? I mean, that happens every 

11 day. That happens to my firm. We upgrade a switch 

12 router. We're not able to connect. Since we're not 

13 allowed to send any orders, the harm is really de 

14 minimis, right? But that was an error, right? We didn't 

15 test the switch software appropriately or sometimes the 

16 exchange makes a change, et cetera, et cetera. That 

17 happens all day every day, right? 

18 So I think as long as those errors are 

19 contained to things about not doing bad things, that 

20 seems appropriate to me. What we're really trying to 

21 focus on is software that is behaving inordinately badly, 

22 and my experience has been that is not software that's 

23 just rolled out. That's software that's been in 

24 production for a while and then hits the magic case that 

25 wasn't tested and goes nuts, right? 
0082

1 I mean, that's the case with the IPOs, right? 

2 That's even the case with Knight to some extent. That 

3 wasn't new software. In my experience every time I have 

4 a bug that causes my firm a loss, it's always something 

5 that's been out for a while, right? It has gone through 

6 all the testing, hit the weird case. Somebody sent a 

7 zero price order that reflected off of something else. 

8 You know, it's two failures. So I just wanted to make 

9 that point. 

10 MR. R. COOK: So, Tom, do you have a question? 

11 MR. BAYER: Yes. So my follow-up question is 

12 regarding once you've introduced software and you've got 

13 defensive code and other monitoring systems that are 

14 available to you, do you go back in and retest your code 

15 after it has been deployed? And what are your software 

16 testing mechanisms to enable you to do that? 

17 So again, if you could talk about defensive 

18 code and how you're modifying your monitoring code 

19 according to changes that have occurred inside of the 

20 marketplace or your use cases if you're talking 

21 technology framework. 

22 MR. PASTINA: If I could just jump in for a 

23 second, so quality assurance teams should be imbedded 

24 with the specification teams because that's where they 

25 build their use cases, and that's where they build their 

0083

1 test cases. And those test cases can reach the thousands 

2 of test cases, and for all of us they're all automated 

3 today. 

4 And every time a new piece of software gets 

5 rolled out, it's not just the new functionality that gets 

6 tested, but that software is regression tested against 

7 all of those old use cases. So I think that's the way 

8 that happens today mostly. 

9 Defensive code you try to build in as best you 

10 can. Unfortunately sometimes you don't recognize that 

11 you need the defensive code until you've had the 

12 situation where you realize now that's where I need a 

13 defense. And so sometimes it comes to them late in the 

14 game. 

15 MR. ISAACSON: And I would just say similar to 

16 what Lou said, you know, we actually at BATS have more 

17 than three times as many lines of code that have to do 

18 with unit testing as we do actual lines of code in our 

19 matching engine. So and that full suite of unit tests 

20 never decreases. It only increases. So all previous 

21 functionality is tested on every single release we put 

22 out in addition to the new functionality. 

23 And we have test driven development that says, 

24 okay, this feature doesn't exist. Write a test that 

25 proves it doesn't exist. Then write the functionality, 

0084

1 and then prove that it passes the test. I mean, it's 
2 kind of Computer Science 101 or Software Engineering 101. 

3 But you know, we spend an enormous amount of 

4 time automating those tests and making sure that the 

5 regression never decreases. 

6 MR. BERMAN: Could we talk a little -- could I 

7 just ask a question about that level of testing, getting 

8 back to something that Jon said? 

9 You're doing unit testing. You're doing 

10 regression testing. We all have virus software on our 

11 computers. My virus software has never caught an old 

12 virus ever because it's always the brand new virus that 

13 came out. There's no definition, and then you get the 

14 download from wherever and then, of course, it never hits 

15 again. 

16 So you build up this massive, massive suite and 

17 now my computer is ten times slower because it's checking 

18 for viruses that never will occur. So I'm quite 

19 sympathetic to the idea that a lot of it is not new 

20 software but old software. 

21 Now, there's a parallel in finance that we've 

22 seen. In 2006, you never tested whether or not a 

23 mortgagee would not pay because all mortgagees paid. 

24 There's no such thing as a credit risk in a mortgage, and 

25 then a few years later we find out that that was just a

0085

1 very bad assumption, and we now have massive amounts of 

2 stress testing where you have independent personnel 

3 saying, "What can go wrong that is independent of the 

4 portfolio manager, independent of the traders?" 

5 But really it's really a unit that's almost 

6 like a SWAT team. People who are thinking about what to 

7 stress, what can possibly go wrong that's really 

8 independent of the process, does that exist at your 

9 firms? I mean, who do you bring in to say, "I'm actually 

10 not part of the development team. I'm not part of the 

11 creating the new order type. I sit around all day and I 

12 think about this could break if the following things 

13 happen"? 

14 If the lights flicker at the New York Stock 

15 Exchange at the exact instance that a car rumbles by, 

16 this fiber optic is going to pull out and that's going to 

17 cause a problem. 

18 MR. ARYA: I think what we thought at our firm, 

19 the way we approach it is not by having somebody in a 

20 separate room who is basically charged with looking at 

21 those error conditions. We tried that to an extent by 

22 having a, quote, unquote, separate QA department way back 

23 when. What we realized is a separate QA department 

24 that's really not living and breathing the operations and 

25 understands the code and its interactions with the myriad 

0086

1 processes is actually not capable of really feeding into 

2 what kind of issues can happen. 

3 So what we do is we take the monitoring and 
4 administrative staff in the firm that are watching these 

5 systems continually operate and administer. They are the 

6 ones that are seeing errors happening. They're the ones 

7 who are seeing -- as a matter of fact, if you go talk to 

8 some of the monitoring staff, they will say when lights 

9 go off and others go off, they have a pretty good idea 

10 while it's that market participant's that typically goes, 

11 does this and this happens. 

12 So folding multiple entities in your firm, 

13 product management, administrative staff that sees these 

14 errors all day long from other sides of the industry, as 

15 well as your QA and desk staff, and coming up with these 

16 test plans that are reviewed in connection with all these 

17 people giving their feedback is extremely important. 

18 So I think the answer is integrating these 

19 teams and giving feedback. That's how we approach it. 

20 MR. NAZARALI: Yeah, we would strongly agree 

21 with that. You know, in our experience, it's not very 

22 helpful having someone from the outside come into the QA 

23 that doesn't really understand the system and how things 

24 work. 

25 What's more important is actually having people 

0087

1 that are battle tested, that have seen a number of errors 

2 happen before, and when they're putting something in, 

3 they remember this hedge condition that caused this other 

4 problem and they know to look for that. 

5 So in our experience it's much, much more 

6 important to have people that really understand the 

7 business, that represent different components, the 

8 trading, the risk management, and that have seen a number 

9 of problems in the past and are able to reflect upon 

10 their experience and draw on their experience to mitigate 

11 the probability of that happening. 

12 MR. BERMAN: Now, can those people be somewhat 

13 independent, meaning so they have that expertise but they 

14 might be from another firm or from a third party, or is 

15 it actually you really have to be imbedded in this 

16 particular software. You may have seen those errors at 

17 another firm, but until you've been battle tested at this 

18 new firm with a new piece of software, you're not going 

19 to be able to contribute in that same way. 

20 MR. NAZARALI: Yes. You really have to be 

21 within that firm because it's so complex that, you know, 

22 being from the outside, you're actually going to draw 

23 resources away from the guys building it because rather 

24 than spend that time testing or implementing it, they are 

25 going to spend all their time explaining to you how this 

0088

1 works and going through code by code, and it's really 

2 ineffective. 

3 MR. LAUER: Can I just say while it's 

4 definitely important to have input from the people who 

5 are building software, frankly, if we're talking like 
6 Computer Science 101, Software 101, the most fundamental 

7 rule is that quality assurance is an independent 

8 function, and one of the gravest mistakes that software 

9 developers make is believing that they can adequately 

10 test their own software. 

11 So while it is important as always to get input 

12 from the software developer and business groups, 

13 independent quality assurance, it's a well understood 

14 principal in software engineering and software 

15 development, that that needs to be independent. It's a 

16 problem because it can be very difficult to attract the 

17 right kind of people into the quality assurance role. 

18 That's probably one of the more difficult things, and why 

19 there exists these problems, is getting the right people, 

20 the battle tested people to want to become quality 

21 assurance. 

22 It's not as sexy. The money isn't as good. It 

23 can be thought of as boring, but it's such an important 

24 role that one possible idea would be to have independent 

25 quality assurance groups within firms mandate a 

0089

1 securities registration, for example, because that gives 

2 them both the compliance perspective, a market 

3 functioning perspective, and means that they should be 

4 paid a little more money. 

5 So you can start to entice the right people 

6 into those independent groups. 

7 MR. R. COOK: I just want to make sure we hear 

8 from --

9 MR. RIGG: No, I was just going to say I think 

10 that as most people have said here, I would see that 

11 across our clients, most clients certainly in this 

12 industry and other industries that have similar 

13 characteristics, the ability to have testers who have the 

14 right level of skill to effectively play their role is a 

15 constant challenge in this space.

16 Obviously in some other industries where you 

17 have a longer development cycle and you can rely upon 

18 higher degrees of documentation, more sets of tools, then 

19 you see independent testing groups or all sorts of 

20 testing groups, all things that can be effectively used. 

21 But in this industry it's very rare, but I 

22 think the challenge though is then as we've seen in some 

23 of these scenarios, is then who in the chain of command 

24 is involved in the decision to take something into 

25 production that doesn't have an interest in that code 

0090

1 making it into production. So often times obviously 

2 you've got the business and you've got technology, and 

3 they're working together. But it's in both of those 

4 groups' interest to move the code forward because 

5 ultimately they both have the same desire, which is to 

6 get that piece of code into production because it usually 

7 represents some feature that they anticipate is going to 
8 increase their market share, drive revenue, et cetera. 

9 So creating that independent check is one of 

10 the challenges. 

11 COMMISSIONER GALLAGHER: So you're talking 

12 about the risks sort of intra-firm, whether it's an 

13 exchange, an ATS, broker-dealer. So just to shake it up, 

14 seeing as you guys are all about the secret sauce of what 

15 you do, what role could peer review play in this space?

16 MR. ARYA: I'd say that I think before we go to 

17 other firms for peer review, just real quick on the QA 

18 side of things, I'm all for autonomy for QA. I think QA 

19 should have an autonomous role to say yes or no in 

20 release of software. 

21 That having been said, I was really talking 

22 about having this autonomous body imbedded within 

23 development, within the guys who understand P&L and risk. 

24 So those go hand in hand. 

25 However, the QA guys say, you know, "We've all 

0091

1 consulted with you, but we don't think this should go." 

2 They should have the final say. 

3 So peer reviews, I think one of the best 

4 practices that we follow quite a bit is peer code reviews 

5 within the firm, getting guys even from other teams or 

6 within the team and actually having the right set of eyes 

7 reviewing the code, reviewing the practices, as Jamil 

8 said, really learning from your previous errors. 

9 There's a huge amount of built in IP in the 

10 heads of all the people who have seen these failures, and 

11 when these developers or QA guys or business guys review 

12 your scenarios, that's huge. 

13 As far as the industry peer review is 

14 concerned, I think there are problems like metrics and 

15 parallel testing and how to build better assimilators. I 

16 think we're better off coming together and having peer 

17 reviews of how to solve those problems vis-à-vis peer 

18 reviews of code and scenarios. 

19 MR. PASTINA: I think peer reviews are 

20 excellent, by the way, within the firm. I think they're 

21 terrific because colleagues, technologists within a firm 

22 are very collegial that way, I find. 

23 I also think it's healthy periodically to 

24 measure yourself with an independent outside group, and 

25 there are several capability maturity models that you can 

0092

1 measure yourself against to see am I a one, am I a five; 

2 where do I rank on the scale? Am I getting any better? 

3 Am I being consistent in my process? 

4 I think that's a healthy thing to do 

5 periodically. 

6 MR. ISAACSON: I think there's just a level of 

7 independence. So you kind of start at the most granular, 

8 which is peer reviews, which we obviously do at BATS and 

9 they are invaluable. You absolutely need to do peer 
10 reviews on anything that's material code. In fact, on 

11 check-ins, we have when you check in a piece of code, you 

12 need to put who reviewed the code. 

13 In addition, independent QA. In order for 

14 independent QA to be effective, they have to know the 

15 markets, and they have to know your system. So for them 

16 to be outside of the firm, you have to pay them an 

17 inordinate amount of money in order for them to know your 

18 system well enough. They can probably learn the markets 

19 well enough, but it's going to take months for them to 

20 know the system. 

21 So having them report independent of software 

22 development or technology but to operations, I think, 

23 makes a lot of sense. That's the way it's set up at 

24 BATS. 

25 And then as far as independents outside the 

0093

1 firm, you know, we do have ARP in place which, similar to 

2 NYSE, we've always taken ARP recommendations as mandates, 

3 not just as recommendations. 

4 In addition to all of our internal and external 

5 audit functions where they're reviewing our software 

6 development life cycle, and you know, I pick different 

7 parts; so I believe there is independent audit in place 

8 today that could potentially be beefed up, but I think 

9 there should be independent audits at many different 

10 levels. 

11 MR. R. COOK: I think Craig had a question he 

12 wants to get in there. 

13 MR. LEWIS: Yes. Thank you, Robert. 

14 So we've heard the perspectives of everybody 

15 and how they approach sort of risk mitigation activities 

16 within your particular entity. Yet your business models 

17 all require you to act in highly interdependent systems, 

18 and so I would like to ask a question that's related to 

19 how do you think about developing sort of system-wide 

20 best practices, and I'll put it in a particular context, 

21 and that is if you think about the demand for speed, it 

22 really drives creating different order types, and to a 

23 certain extent those order types underlying that reflect 

24 essentially nonlinear trading strategies. 

25 So the question I have for you is if we're 

0094

1 worried about complexity, where is the best place to 

2 address complexity? Would it be at the exchange where we 

3 would create a complex order type that may be harder to 

4 code, or should we push that off to the entity and 

5 essentially require them to create an algorithm that 

6 allows them to execute a nonlinear trading strategy in 

7 real time? 

8 How do you weigh those two risks? 

9 MR. ROSS: I actually think that it's not quite 

10 that easy. I mean, if you talk about high frequency 

11 trading or speed, the faster you go -- I mean, this makes 
12 sense -- the less you do, right? If you want to do a 

13 lot, if you want to do a mark-off simulation, you're not 

14 going to do it in real time. It's just not going to 

15 happen. 

16 So the faster you go, the simpler you get. So 

17 with that being the premise, I don't think these order 

18 types are driven by high frequency trading. As a matter 

19 of fact, high frequency trading actually prefers a 

20 smaller number of simpler order types, and that's 

21 historically how they prefer to function. 

22 I see the order peripheral types of --

23 proliferation of order types in the market not driven by 

24 high frequency trading. I think they're driven by the 

25 market centers themselves, to some extent how market 

0095

1 centers need to react to each other, the reactions from 

2 Reg. NMS, and you see actually some of those going away, 

3 some of those early order types that were done just after 

4 2007, post Reg. NMS, actually disappearing now because 

5 they've come up with simpler ways to implement their Reg. 

6 NMS obligations. 

7 So I think that whole premise is actually not 

8 correct. I think the basic premise is the faster you 

9 want to go, the simpler you need to be. 

10 MR. NAZARALI: I'd like to add a point. I 

11 think the idea of high speed trading captures the 

12 imagination. You think about these co-located boxes. 

13 You think about the fact that the cables have to be equal 

14 length between the network and each of the boxes. You 

15 think about the high speed line between Chicago and New 

16 York. 

17 It makes a lot of great headline news for the 

18 newspapers. If you look at most of the automated trading 

19 firms, most of our effort is spent on developing trading 

20 strategies to add liquidity to the marketplace. It's not 

21 really on getting a little bit faster. 

22 Yes, that's a small part of it, but it's 

23 relatively small compared to the amount of publicity it 

24 gets. 

25 MR. LAUER: I think the order type question is 

0096

1 a very good one and an important one. It is the 

2 proliferation of all these order types and the complexity 

3 of these order types that is adding unnecessary 

4 complexity to the market, which is already an extremely 

5 complex system as it is, and like I said earlier, not 

6 very well understood even by the most advanced 

7 participants, especially at how these different complex 

8 systems interact. 

9 Not only that, when you have complex order 

10 types, it leads to extremely complex testing scenarios, 

11 and you are not going to pick up all the things you could 

12 or should because you don't know what that actual 

13 matching engine logic is in general. 
14 I think that order types should be revisited. 

15 There are so many of them now. There should be ample 

16 evidence as to the utility of every order type. 

17 An Exchange should be able at this point to say 

18 well, we have had all these order types in place for a 

19 very long time, and here's all of the data that we can 

20 show to demonstrate the utility of these order types to 

21 the long term investor, and if that isn't demonstrated, 

22 maybe some of these order types could go away and we 

23 could have a greater drive towards simplicity, which 

24 would make testing a simpler exercise, which would 

25 improve investor confidence in different ways as things 

0097

1 simplify rather than add complexity. 

2 On the high speed trading side, while generally 

3 many high frequency traders are adding liquidity to the 

4 market, it was found that especially during times of 

5 market stress and the flash crash that they became market 

6 takers and exacerbated the problem. 

7 It's a double edged coin and one that I don't 

8 think is as simple to pin down as it could be. 

9 MR. ISAACSON: I just wanted to make a point on 

10 the order types. We are all well aware that order types 

11 go through the normal rule filing process and usually a 

12 long comment process. 

13 A lot of the order types that have been 

14 introduced are a result of Reg. NMS, frankly. Price 

15 sliding order types, things like that, to avoid locking 

16 and crossing ISOs, ISOs that deal with the trade through 

17 rules. The order type is directly related to one or many 

18 regulations, whether it be Reg. NMS or a regulation 

19 before that. I think that is just the Exchanges 

20 responding in an effort to be compliant and offer for our 

21 members the tools they need to trade. 

22 In addition, I think the vast majority of order 

23 types frankly has to do with routing strategies, 

24 Exchanges routing to each other. I can't speak for the 

25 ATSs or the market makers, but Exchange order types, we 

0098

1 have a lot of routing order types based upon member 

2 demand. However, the typical high speed trader, HFT, 

3 that people think of, very rarely uses an Exchange 

4 routing order type because they have linkages to all the 

5 Exchanges themselves. They don't need our smart order 

6 router. It is usually someone who doesn't necessarily 

7 have connectivity everywhere. 

8 A lot of the order types, as Jamil said, are 

9 not really focused -- and Jon said -- are not focused on 

10 high speed traders. They're focused on people that want 

11 a greater suite of functionality that maybe they don't 

12 have at their fingertips within their firms. 

13 MR. LAUER: One particular order type that has 

14 recently been introduced is the PL Select order type. It 

15 seems the only explanation is for high speed traders. 
16 That's an example of one that could be revisited as its 

17 only purpose is that it doesn't interact with ostensibly 

18 professional flow. 

19 That again is an example of something that 

20 seems to be adding unnecessary complexity and leads 

21 people to trust the markets that much less. 

22 MR. ARYA: I would say in isolation, most of 

23 the order types made sense, but going back to the testing 

24 point and integration testing, the whole suite of order 

25 types, if you will, actually presents a pretty huge 

0099

1 challenge for us to actually test and so forth. There is 

2 always a demand for those order types when they come out, 

3 so we are required to really comply and also test. 

4 I think the collection of ever growing order 

5 types requires a review, and as somebody else said, if we 

6 could review really what is the amount of volume and 

7 utility of those order types, I think that is wise to do. 

8 COMMISSIONER GALLAGHER: Chris raises a really 

9 good point, which is very important to me, which is the 

10 extent to which regulations has driven some of the 

11 technology issues we're dealing with today versus market 

12 practices. 

13 I think Dave raises an order type that is 

14 peculiar to high frequency trading. Chris, we have seen 

15 others. I remember years ago the New York Stock Exchange 

16 "do not ship" order type was right after NMS. 

17 I think it is incumbent upon us, too, as we sit 

18 here with you, we are asking you to think deep thoughts 

19 and help us better oversee these issues. 

20 You need to point out to us and not be shy 

21 about where we need to think about our regulations, our 

22 interpretations, our FAQs, whatever it is if we are 

23 driving this type of behavior, which isn't healthy for 

24 the markets in some way, we need to know about it. We 

25 need to address it. 

0100

1 MR. PASTINA: That DNS order type remains the 

2 most popular order on the Exchange today. 

3 COMMISSIONER GALLAGHER: That is the funniest 

4 rule filing I've ever seen. 

5 MR. PASTINA: It was directly because of Reg. 

6 NMS and for people who wanted better control of their 

7 orders because they had decision making technology 

8 upstairs, so they did not want to be shipped to other 

9 markets. As Chris said, most of these new order types 

10 are in reaction to market structure changes and 

11 competition. 

12 One of the interesting things, I think, about 

13 stability is the more change you introduce into the 

14 environment, the more opportunity there is for something

15 to go wrong. One of the things about market structure 

16 and its continuing evolution is that as it continues to 

17 change and that change increases and grows more rapid, we 
18 all will react to that, and all of the technologies have 

19 to be adjusted and they all have to be tested. 

20 At some point, there has to be a deep breath as 

21 to how much change is actually being introduced into the 

22 system. 

23 MR. ISAACSON: I just think when the SEC put 

24 out the concept release prior to the flash crash, we at 

25 BATS thought that was an excellent 73 pages of very 

0101

1 thoughtful questions, frankly. It hasn't got a whole lot 

2 of air time because of the events of May 6 and Dodd-Frank 

3 and all those things. 

4 We have had NMS in practice for five years. It 

5 was approved seven years ago. Now is probably a good 

6 time to raise that concept release again and to include 

7 obviously capital markets are almost 100 percent driven 

8 by technology, is there a way to simplify things. 

9 One of the mantras at BATS is keep it simple. 

10 We believe we cannot operate efficiently over time 

11 unless we get rid of stuff. We cull stuff that is not 

12 used. As a whole market, I think we believe that as 

13 well. I think we should do it holistically, answering 

14 many of the questions that the concept release raised. 

15 MR. BURNS: Following up on the point that 

16 Chris raised and Commissioner Gallagher pointed to, right 

17 now we do have the ARP program in place coming into the 

18 Exchanges, some of the ATSs, and think about it at least 

19 now acting in some respects as an independent or across 

20 Exchange review.

21 You talked about ways of beefing up that 

22 program, and I guess as it relates to testing or error 

23 prevention, are there concrete take away things we ought 

24 to be contemplating to focus ARP and its efforts in a 

25 risk oriented way. 

0102

1 MR. ISAACSON: I have to commend the ARP staff. 

2 I believe as we became an Exchange in 2008, we have seen 

3 their review of us become more rigorous over time, and 

4 their focus has become much more intense. I have to 

5 commend the staff in that regard. 

6 Regarding beefing up, it was not necessarily 

7 regarding ARP. It was independent or external audit 

8 review, which we have been through a pretty intense 

9 process of post-RIPO issues. 

10 I can't really give specific things there other 

11 than I commend the efforts of the staff and the 

12 Commissioners, Gregg, especially getting the Midas system 

13 installed, understanding how the market interacts even as 

14 a stop gap until we have an audit trail. I think that is 

15 fantastic, the more technology and folks you have on 

16 staff that could come in and audit us. 

17 As we were talking about QA, the utility of QA 

18 is only as good as the people that understand the markets 

19 and your system. ARP needs to have people that can come 
20 in and understand our system at a very deep level. 

21 I'll be honest, when we hire a software 

22 developer, I figure it's going to take him six months 

23 before he provides any utility to me, no matter how good 

24 he is, because there is just a lot of complexity. There 

25 is a lot to know in the market. We require developers to 

0103

1 get licenses. It takes a lot for them to get up to 

2 speed. Just focusing the efforts of ARP on understanding 

3 the innards of not just Exchanges but ATSs as well would 

4 be very useful. 

5 MR. ROSS: I think ARP provides a fair amount 

6 of value both to Exchanges and ATSs, and it's not 

7 unreasonable that you would want to see something like 

8 that for participants who are connected to an Exchange. 

9 What I would say is if you're a firm and you're 

10 connected to a broker, third party, not connected to an 

11 Exchange, historically, you only send a few orders a day, 

12 probably don't need the extent of rigor as a firm that is 

13 directly connected to an Exchange, directly regulated by 

14 FINRA and the SEC, standard practice, the more systemic 

15 risk you pose to the system, the more highly you should 

16 be regulated. That seems incredibly appropriate. 

17 That is the approach you have taken for ARP for 

18 Exchanges and it seems reasonable for broker-dealers to 

19 do something relatively similar. The one thing that we 

20 haven't talked about, and I know this is a topic of the 

21 next panel, is operations. There is very little 

22 literature in the finance market for best practices 

23 around operations. 

24 Operations is really where the rubber hits the 

25 road, right. You know, everyone who has worked at a firm 

0104

1 knows a good operations guy when they see them. It's 

2 kind of hard to define exactly what that is, right. 

3 Operations, their whole job is to deal with 

4 conflict, conflict of interest. They are running a 

5 router that is routing out order flow for four desks and 

6 a bank, they see something that goes down, their ability 

7 to shut off their system and not face getting their feet 

8 held to the fire from internal P&Ls they have upset, is 

9 critical. That is a culture thing. That is a command 

10 and control thing that is absolutely important. 

11 Even Exchanges, this is somewhat dated 

12 information, but when I worked in an Exchange, my 

13 regulatory department was really hesitant to shut firms 

14 off because of the fair access rules, right. That is a 

15 very straight ahead internal conflict, like Exchanges 

16 must provide fair access. 

17 We didn't have any rules set about when it was 

18 appropriate or not appropriate to take action. It could 

19 be seen that the Exchange is taking action to shut off a 

20 participant of its own accord which is in violation with 

21 fair access. 
22 Clearing up a lot of those internal conflicts 

23 is very important. My point is a really good operations 

24 person navigates those internal conflicts very, very 

25 well, and there is not a lot of best practice literature 

0105

1 around that. I think that would be incredibly helpful. 

2 COMMISSIONER GALLAGHER: On the ARP point, I 

3 hear you on ARP, and this is why I asked about peer 

4 review. I think whatever we do with ARP going forward, 

5 there has to be an assumption that the Government is not 

6 going to ferret it all out and figure it out. We can 

7 hire the smartest people in the world. We will never 

8 have enough, it will never be current enough to keep up 

9 with you guys. 

10 I think pure reliance on ARP isn't going to get 

11 us there. It can really help, it can bring rigor and 

12 some sort of cross pollination of best practices. I 

13 don't want to put too many eggs in one basket. 

14 MR. ISAACSON: Commissioner Gallagher, I would 

15 agree, ARP should not be the only defense or the only 

16 audit. There needs to be external firms as well as rigor 

17 within the firm. 

18 I think peer review amongst different firms 

19 could be useful. However, in practice and 

20 implementation, I think it might be quite difficult for 

21 even the smartest guys at the firms to have enough 

22 context, like if I were to show up at NYSE tomorrow and 

23 Lou lets me see all the code, it is probably going to 

24 take me a few months before I can give any really 

25 intelligent feedback or recommendations on what he ought 

0106

1 to be doing. 

2 MR. ARYA: I think peer participation is really 

3 -- where it comes in handy is if we can identify 

4 problems. One of the questions was about the role of 

5 independent parties and testing and so forth. 

6 I think there are common problems that we all 

7 try to solve, building simulators, market data replay's 

8 and so forth. If we can find ways and discuss metrics, 

9 that's where peer participation is hugely useful, not in 

10 reviewing code and so forth. 

11 MR. LAUER: I think along the lines from the 

12 other perspective, the firms that are doing the trading 

13 and building the software, quality management standards 

14 are something to consider. It is sort of a nice middle 

15 ground between peer review and direct auditing by the 

16 SEC, for example. 

17 CMM is one that has been thrown around and 

18 mentioned earlier. ISO 9000. Firms or industries that 

19 have adopted some form of ISO 9000, aerospace, chemicals, 

20 medical devices, health care, food safety, I'd like to 

21 think financial services is as critical or more to the 

22 well being of the country, and as such, we could look at 

23 any firm that does have direct market access having to 
24 adhere to quality management standards. 

25 I don't think they want to hear that because 

0107

1 that is a substantial cost, and it would drive some 

2 players from the market, but the cost/benefit analysis 

3 should be rather clear, I would think. 

4 MR. R. COOK: With that, I think we are at the 

5 end of the time for our first panel regrettably. 

6 I want to thank the panelists for their 

7 insights and candor. We look forward to reviewing all of 

8 your comments you have submitted in the file. 

9 We now have an hour and a half break for lunch 

10 before we start our second panel. 

11 If you leave the building, please leave your 

12 visitor pass at the front desk, and make sure you allow 

13 enough time to get back through the security screening 

14 upon re-entry. 

15 We will get back together and begin the second 

16 panel at 2:00. Thank you. 

17 (A lunch recess was taken.) 

18 A F T E R N O O N S E S S I O N 

19 RESPONDING TO ERRORS AND MALFUNCTIONS 

20 AND MANAGING CRISES IN REAL TIME 

21 MR. R. COOK: I am going to get us started now 

22 on our second panel of the day. Welcome back, everyone. 

23 This panel is entitled "Responding to Errors 

24 and Malfunctions and Managing Crises in Real Time." As 

25 the title suggests, this panel will cover error response, 

0108

1 focusing on how the market might employ independent 

2 filters, objective tests, and other real time processes 

3 or crisis management procedures to detect, limit and 

4 possibly terminate erroneous market activities when they 

5 occur. 

6 I think one of the themes of the first panel 

7 was while we had a number of very helpful suggestions 

8 about best practices oriented towards the roll out of new 

9 software and how to minimize the risk of error, that 

10 errors will occur. 

11 What we are really focusing on in this panel is 

12 what do we do about that. I think we are particularly 

13 interested in hearing from our panelists about how to 

14 approach that both from a technology perspective but also 

15 in terms of internal organizational control and people 

16 management, and how the people and the machines are 

17 interacting in the space. 

18 We have a great panel lined up today, as we did 

19 this morning. I would like to first begin by asking each 

20 panelist to briefly introduce themselves, after which I 

21 will ask Dr. Markus to start us off with her perspectives 

22 for a few minutes, and then we will jump into some 

23 questions from the staff to kick off the conversation. 

24 David, why don't we start with you? 

25 MR. BLOOM: Good afternoon. Thank you to 
0109

1 Chairman Schapiro and the other Commissioners and staff 

2 for inviting us to the panel. 

3 My name is David Bloom. I'm the head of Group 

4 Technology for the Americas for UBS. UBS is present in 

5 all major financial markets worldwide with offices in 

6 more than 50 countries, about 36 percent of our workforce 

7 is in the Americas where I am responsible for our 

8 infrastructure. 

9 We execute large volumes of equities on behalf 

10 of our customers with number two market share on NASDAQ 

11 for 2011 and number three market share on the New York 

12 Stock Exchange for 2011. 

13 We are honored to have a seat on the panel. 

14 Thank you again for having us. I would also just point 

15 out that we were one of the signatory firms to the 

16 comment letter mentioned earlier this morning. 

17 MR. C. COOK: Chairman Schapiro, Commissioners 

18 and SEC team, thanks for having me here. This is a 

19 really important step, I think, to elevate the importance 

20 of technology and how we manage it. 

21 My name is Chad Cook. I am the Chief 

22 Technology Officer of Lime Brokerage. Collectively, we 

23 touch over ten percent of the U.S. equity volume more or 

24 less on a daily basis. 

25 Lime Brokerage, we have been around for ten 

0110

1 years, for over ten years. We specialize in high 

2 performance trading systems, and with that, we built in a 

3 lot of risk technology from day one. 

4 I think our approach is somewhat unique in that 

5 respect and we arrive at our technology looking at it 

6 from good technology practices, as has been discussed in 

7 the previous panel, but also from a risk management 

8 approach coming from our backgrounds in mission critical 

9 systems and datacom and IT security. 

10 My background, I have over 20 years building 

11 technology, primarily in datacom and IT security. 

12 We apply a lot of these architectures and 

13 models for managing and operating technology to the 

14 financial sector, which has been pretty interesting and 

15 engaging for us. Thanks. 

16 MS. EWING: Good afternoon. My name is Anna 

17 Ewing. I'd like to thank Chairman Schapiro, 

18 Commissioners, and the rest of the SEC staff, for having 

19 us here today. I'm delighted to be able to participate 

20 and really look forward to continuing the dialogue from 

21 this morning that I felt was very engaging. 

22 I'd like to give you a little bit of my 

23 background. I'm the EVP and CIO at NASDAQ OMX. I have 

24 responsibility for the Global Technology organization, 

25 and that is all of the software development, 

0111

1 infrastructure, technology operations, and information 
2 security functions. 

3 I'm also the head of the Market Technology 

4 business unit, and that business unit provides commercial 

5 technology to 70 Exchanges and markets around the world 

6 dispersed over 50 developed and emerging countries. 

7 I've been at NASDAQ OMX for 12 years, and I 

8 have been part of the transformation of NASDAQ from a 

9 U.S. cash equities market to who we are today, which is a 

10 global diversified enterprise providing operations to 25 

11 markets here in the U.S., and in Europe. We also have three 

12 clearing houses and five central security depository 

13 organizations that are part of our organization. 

14 Before NASDAQ OMX, I have been in financial 

15 services, really my whole career, for 25 years. I've had 

16 the privilege of living through the evolution of this 

17 industry. I've had positions in both CIBC World Markets 

18 and at Merrill Lynch. 

19 I would like to say in NASDAQ OMX's view, 

20 history clearly shows that technology has helped 

21 investors. Automation has increased transparency, lowered

22 trading costs, and offered regulators powerful tools to 

23 eliminate bias and favoritism, and to detect and prevent 

24 fraud. 

25 Today's technologies are inclusive, offering 

0112

1 all market participants the opportunity to use these 

2 tools. Technology has been and remains best understood 

3 as a solution and not the problem as we continue today's 

4 dialogue. 

5 Thank you. 

6 MR. GAMBALE: Good afternoon. Let me thank 

7 Chairman Schapiro, Commissioners and Commission staff for 

8 inviting me to the panel. 

9 I'm Albert Gambale. I'm the Chief of 

10 Applications Development for the Depository Trust and 

11 Clearing Corporation. DTCC is a key player in the 

12 industry infrastructure. We are the central securities 

13 depository for the United States. We have run a number 

14 of clearing corporations, and we are a designated 

15 critical member of infrastructure for the U.S. 

16 Personally, I've been in the securities 

17 industry, all in IT, for 30 years. The last 15 years 

18 have been at DTCC. Thank you. 

19 MR. JAHANI: Madam Chairman, members of the 

20 SEC, participants and members of today's SEC panel, 

21 please allow me to first of all thank you all for giving 

22 us the opportunity to participate in this roundtable. 

23 My name is Saro Jahani. I am the CIO of Direct 

24 Edge, probably the newest and youngest Exchange in the 

25 equities market in the U.S. Direct Edge's market share 

0113

1 basically has raised from one percent around 2007 to 

2 almost ten percent nowadays. We have basically shown 

3 more than anything that this type of rise would never 
4 have occurred unless there were innovational and 

5 structural changes in the U.S. that has helped us to get 

6 where we are. 

7 We are very confident that this panel is coming 

8 at the right time. It is very hopeful on behalf of 

9 Direct Edge to believe it is going to take us to points 

10 where we will be able to come up with some great ideas. 

11 Personally, I've been in the area of 

12 information technology for almost 25 years, both in 

13 Sweden, where I grew up, and also here in the U.S. 

14 I am hoping we will be coming up with some good 

15 ideas that will help us to innovate further. Thank you, 

16 once again. 

17 MR. STEINBERG: Chairman Schapiro, 

18 Commissioners, SEC staff, thank you for inviting me. 

19 My name is Lou Steinberg. I'm the Chief 

20 Technology Officer for TD Ameritrade. TD Ameritrade 

21 serves roughly 5.7 million retail investors who execute 

22 typically 300,000 to 400,000 trades a day. 

23 We run a very un-conflicted model in that we 

24 are really focused on retail investors. We don't do a 

25 lot of high frequency trading and that kind of stuff. 

0114

1 As CTO, I run technology platform engineering 

2 and development. I run technology operations and I run 

3 information security and fraud. 

4 Previously, I had a software company in the 

5 operational risk management space focused on high 

6 availability, and over the last 25 years, I have built 

7 and run a number of software start-up's. 

8 MR. R. COOK: Thank all of you. We really 

9 appreciate you taking the time today to share with us 

10 your insights. 

11 This morning we introduced who else was around 

12 the table, so I'm not going to go through that again, 

13 other than to note we have Dawn Patterson with us now from 

14 the Office of Compliance, Inspections and Examinations, 

15 so welcome. 

16 Also a special welcome to Andrei Kirilenko, who 

17 is the Chief Economist at the CFTC, and from the future 

18 side of the world, and we welcome him and invite him to 

19 ask questions as they might arise. 

20 As I mentioned, we are going to ask Professor 

21 Markus to lead off. I will take the liberty of re22 introducing her because some of you might not have heard 

23 her introduction this morning. 

24 Dr. Markus is the John W. Poduska, Sr. 

25 Professor of Information and Process Management at 

0115

1 Bentley University. She is also a research affiliate of 

2 MIT's Sloan Center for Information Systems Research. 

3 Professor Markus' teaching, research and 

4 consulting interests include enterprise and inner 

5 organizational information systems, the unintended 
6 consequences of information technology and risk 

7 management strategies, and IT in the mortgage industry. 

8 She was named a Fellow for the Association for 

9 Information Systems in 2004, and received the AIS LEO 

10 Award for Exceptional Lifetime Achievement in Information 

11 Systems in 2008. 

12 Thank you very much for joining us today, and 

13 please, why don't you share with us your perspectives? 

14 DR. MARKUS: Good afternoon, all. Unlike 

15 Professor Leveson, I have only 35 years of experience 

16 doing research in the area of organizational information 

17 technology. 

18 My background is in industrial engineering, 

19 organizational behavior, and information systems, which I 

20 think puts me in an interesting position to look at 

21 systems dynamics of the sort we are talking about here. 

22 For the past 15 years, I have been studying the 

23 use of information technology in the mortgage industry. 

24 Much of that research was supported by the National 

25 Science Foundation and was conducted with colleagues, but 

0116

1 the conclusions that I have made and will be talking 

2 about here are really just my own. 

3 After the mortgage crisis, I wanted to learn 

4 what role, if any, information technology may have played 

5 in the crisis. I examined the ways that mortgage 

6 technology has evolved and how it may have contributed to 

7 the risk of errors. 

8 I concluded that information technology did 

9 have a role in the crisis, and what I will first do is 

10 describe briefly what I learned about mortgage 

11 technology, and then I'll talk about what I think are the 

12 implications for trading technology. 

13 I studied the evolution of mortgage technology, 

14 especially automated underwriting, from its introduction 

15 in 1994 until shortly before the housing bubble formed. 

16 That is a very short period of time, only about ten 

17 years. 

18 In that space of time, automated underwriting 

19 technology went from being a discrete innovation with 

20 very small local scope to becoming a highly complex and 

21 interconnected web of systems that spread across many 

22 different organizations, and it touched almost everyone 

23 in the U.S. and many people elsewhere around the globe. 

24 I'd like to talk about the nature of the 

25 changes that occurred in automated underwriting 

0117

1 technology because I think it's important to understand 

2 how technology evolves and how that evolution, while 

3 bringing many benefits, can also contribute to risk of 

4 crisis. 

5 First, the lending process was streamlined as 

6 it was automated. Technology became easier to use, and 

7 it was embedded in loan origination software that was 
8 already being used by lenders. 

9 Fewer inputs were required, fewer supporting 

10 documents were required, and outputs became simpler, too. 

11 Instead of being given long lists of conditions, people 

12 were simply told accept, caution, or refer this for more 

13 analysis. 

14 Because the outputs were simpler, they became 

15 easier for people to act on, and people became more and 

16 more confident over time about the decisions produced by 

17 the system and stopped questioning the quality of those 

18 decisions. 

19 Second, technology was extended to a greater 

20 variety of lending decisions. Automated underwriting 

21 technology was first applied to prime loans. Then it was 

22 applied to jumbo loans, Alt-A, and A- loans. 

23 It was not long after that before technology 

24 was used to make decisions about subprime loans, high 

25 loan to value loans, low down payment loans, loans for 

0118

1 manufactured housing, construction loans, commercial 

2 property loans, HELOCs, ARMs, option ARMs, cash out 

3 refinancing's, and reverse mortgages. 

4 At the same time that the technology was being 

5 extended to many other types of decisions, the technology 

6 was also accepting a greater proportion of loan packages 

7 that were submitted to it. 

8 Third, the technology was extended to a greater 

9 range of lending tasks. The technology first just 

10 evaluated borrower's creditworthiness. Then it performed 

11 automated property appraisals. Then it expanded into 

12 loan servicing. Then it expanded into mortgage 

13 securitizations. 

14 At each expansion, the systems developed by one 

15 organization were interconnected with systems that were 

16 developed by other organizations. 

17 At each expansion, new user groups began to 

18 interact with automated underwriting technology. 

19 At the end, it was not just mortgage lenders, 

20 underwriters and credit agencies that were working with 

21 automated underwriting, it was also brokers, real estate 

22 agents, and many individual consumers who used on line 

23 lending websites. 

24 Fourth, in the end to end mortgage process, 

25 which spreads across many different organizations and 

0119

1 individuals, it became restructured as a result of all 

2 these other changes. 

3 No individual or organization could see the 

4 whole picture. Some human oversight was eliminated from 

5 the process in part because people assumed that there 

6 were check points elsewhere. Fraudsters found ways 

7 around the controls. 

8 It is important to understand that this 

9 evolution of technology didn't occur in just one place. 
10 It was spread across many different organizations. Each 

11 organization independently tested its own software. When 

12 the systems were interconnected, the connections were 

13 tested. 

14 The systems continued to evolve and they 

15 evolved rapidly and in many different ways, even after 

16 the connections were made and tested. 

17 In summary, automated underwriting technology 

18 in the mortgage industry evolved from a simple discrete 

19 technology with few interconnections into a very complex, 

20 tightly coupled system that touched and was touched by 

21 many different people, processes, and organizations. 

22 I believe that this speeded up the flow of 

23 mortgage lending in a way that fueled both the housing 

24 price bubble and speculation in mortgage backed 

25 securities and derivatives. 

0120

1 What does this all mean for trading technology? 

2 I understand that trading processes and technology 

3 differ from those in the mortgage industry in a number of 

4 ways. 

5 Crucially, trading technology is even more 

6 complex and interconnected than mortgage technology. 

7 Consequently, I believe that technology is even more 

8 vulnerable to errors, disruptions, and crises in trading 

9 and markets than in the mortgage process. I think the 

10 events of the last few months are the new normal. 

11 When systems are complex and tightly 

12 interconnected, I think there are about four basic 

13 strategies for preventing and limiting damage. The first 

14 is to reduce system complexity and interconnectedness by 

15 design, simplifying and de-coupling. 

16 Second, testing systems thoroughly beforehand. 

17 Third, monitoring effectively in real time and reacting 

18 quickly to errors that arise, and then increasing 

19 capacity to recover quickly when fast reaction is not 

20 enough. 

21 I think many of the panelists this morning and 

22 also I'm sure the panelists this afternoon and many of 

23 the people who posted comments on the website have come 

24 up with individual strategies that fall into one of those 

25 categories or another. 

0121

1 I think the most important point is that in 

2 order to assure a robust system that protects the 

3 interests not only of investors but all citizens, that 

4 these four strategies need to be pursued simultaneously. 

5 Just picking one strategy and focusing on that is not 

6 really going to address the problem. 

7 Let me just briefly reiterate those strategies 

8 and mention a few of the many solutions that I've heard 

9 other people raise. The first strategy is to reduce 

10 system complexity, and some of the suggestions have been 

11 things like restricting trading of assets to one trading 
12 system with managed gateways to other systems. 

13 Another was to slow down and batch trades in 

14 high frequency trading, and I'm sure there are a number of 

15 other solutions that would fall into this category of 

16 reducing system complexity by de-coupling and inserting 

17 buffers. 

18 The second strategy is to test systems

19 beforehand, but I'd like to emphasize here that it's 

20 extremely important to do that, but testing alone inside 

21 individual organizations is not enough. 

22 It's important to conduct tests on an end to 

23 end basis. I understand how difficult it could be to do 

24 this well, but I believe it's very important to do it. 

25 Why is it difficult? Let's think about this. 

0122

1 If we were to have done an end to end test of mortgage 

2 technology software, we would have had to bring together 

3 agents, brokers and borrowers, some of whom were 

4 fraudulent on the one hand, and on the other hand, we 

5 would have had to have investors in mortgage securities. 

6 It's not just the lenders and the securitizer's that are 

7 affected. Also, you would have had to include potential 

8 hackers and cyber terrorists if you were to do a complete 

9 end to end test of that system. 

10 I believe that it is important to work very 

11 diligently at end to end testing, and I think that 

12 probably will require much greater industry-wide 

13 participation than currently occurs. 

14 Tests must occur regularly, and I think live 

15 simulations that combine problem triggers that are known 

16 from past events, along with creative new scenario's like 

17 denial of service attacks, are needed in order to do the 

18 tests. The tests would also, I believe, have a side 

19 benefit. They would build skills in real time error 

20 detection and quick recovery. Simulation events like 

21 this are common in public safety, and I believe they have 

22 a place in the financial services industry as well. 

23 The third strategy is to monitor and react in 

24 real time. Many of the suggestions have dealt with ways 

25 of monitoring transaction flow and developing better 

0123

1 circuit breakers and kill switches. These are examples 

2 of that third strategy. 

3 The fourth strategy is to improve capacity, to 

4 recover quickly, and I know that's a large part of the 

5 panel this afternoon, so I will say no more on that. 

6 In conclusion, I believe complexity and 

7 interconnectedness of trading technology is such that 

8 errors, disruptions and crises may be the new normal. 

9 To reduce the negative consequences of these 

10 events for investors and all citizens, I believe the four 

11 strategies should be pursued simultaneously, reducing 

12 complexity, conducting end to end tests, monitoring and 

13 reacting in real time, and increasing capacity to recover 
14 quickly. 

15 Thank you. 

16 MR. R. COOK: Thank you, Professor Markus. I 

17 especially appreciate you reviewing the comments that we 

18 have received in the comment file, which remains open, 

19 and we encourage anyone who is interested in submitting 

20 their comments for consideration by the Commission to do 

21 that. 

22 Bridging your experience and analysis in the 

23 mortgage industry to the securities industry, I think 

24 that is a nice segue to get into a little bit more

25 granularity about how we pursue some of the four 

0124

1 strategies you were just identifying for us. 

2 Let me ask Gregg Berman to kick us off with the 

3 first question. 

4 MR. BERMAN: Thanks, Robert. Much as we did 

5 with the first panel, I think we want to start off with 

6 the general topic about best practices, but in this case, 

7 best practices for identifying the errors and containing 

8 them as they happen. 

9 I would like to modify that a bit from the 

10 questions you were given, just based on Dr. Markus' 

11 comments about the interconnectedness of the markets, and 

12 we all understand that, but I think she highlighted some 

13 points about the difference in the errors that can occur 

14 internally and those that arise because of the 

15 interconnected nature. 

16 As you think about the best practices, maybe 

17 you can highlight a bit of the type of error checking and 

18 the type of concerns that you have for understanding what 

19 is going on if you have a problem internally versus if 

20 you have a problem that seems to be coming externally, 

21 either from a client or from a system you are hooking up 

22 to, and there might be differences in how you actually 

23 think about each of those things. 

24 Why don't we start at the far end with Lou, if 

25 that is okay. 

0125

1 MR. STEINBERG: Each incident that we have goes 

2 through a number of different stages. We start with a 

3 set of best practices around managing the incident 

4 itself. We happen to use ITEL as the framework, which we 

5 believe helps us organize the problem. 

6 It also gives us a set of run books, 

7 particularly if it looks like an internally triggered 

8 incident, so we know how to deal with it. It gives us a 

9 set of pre-defined actions to take if we believe that the 

10 trigger may have come from an external source as well. 

11 The notion of a trigger is important to us 

12 because there are actually three different components of 

13 an incident in our view. 

14 One is you have a latent defect. The second is 

15 you have a triggering event, and the third is you have an 
16 impact or potential impact which generates something we 

17 call "materiality." 

18 We have put in place a number of things through 

19 ITEL to give us standard responses to pre-allocate 

20 resources, expertise, specialists, to help us de-bug and 

21 manage through the incident, to try to contain it if it 

22 wasn't contained in the design phase through 

23 segmentation, to then try to limit it. 

24 If nothing else, to recover, to stop the damage 

25 and then recover quickly. 

0126

1 The actual steps that we take on an externally 

2 triggered event or with an external latent defect are 

3 probably to separate ourselves from that source. 

4 One option might be that we choose, if we see 

5 some unusual behavior with a market maker, to route to a 

6 different market maker, and we can do that selectively. 

7 If it's internal, we will limit the damage, we 

8 will immediately reset the systems that are affected, 

9 work around the systems that were affected. In some 

10 cases, we use a technique we are building called "fast 

11 fail back," which allows us to effectively rewind our 

12 system's hardware and software to a previously known good 

13 state. 

14 MR. BERMAN: Thanks. Others? 

15 MR. JAHANI: ITEL was mentioned. It is quite 

16 interesting. I was actually hoping there would be a 

17 little bit of ITEL tweaking in today's discussion. 

18 For us, at Direct Edge, we have deployed ITEL 

19 on a broader basis. While everybody wants to sort of 

20 have a crisis situation topic discussed, we are actually 

21 pointing out that everything is actually interconnected, 

22 the very same way basically preventing a problem and 

23 addressing them are also interconnected. 

24 We can't just address them one by one. It has 

25 to be cohesively and realistically. 

0127

1 ITEL has been a great framework for us because 

2 it actually has looked at the process, technology and 

3 people, the three components are there. 

4 We have addressed this since a couple of years 

5 ago by sort of looking at a principle of having an 

6 adaptable model, a sustainable model, repeatable model, 

7 and demonstrative model. Those four components have 

8 helped Direct Edge to come up with a very robust incident 

9 management discipline. 

10 The problem management discipline, incident 

11 management discipline, and change handling and all that 

12 stuff are also interconnected. 

13 We have been able to create what we call a 

14 "Code Blue" model that really is there to help us to 

15 address our issues. In this situation, probably the most 

16 important thing to remember is actually the impact and 

17 nothing else, because we want to address the impact. 
18 In order to address the impact, we can't just 

19 basically look at technology. We also need to have the 

20 compliance component, the regulatory component. By 

21 following a Code Blue discipline as a best practice 

22 model, we have been able to address this issue properly. 

23 Once again, I want to just basically 

24 distinguish that addressing issues and managing crisis 

25 situations has to be focused on market impact, whereas 

0128

1 when we are preventing, you are actually addressing the 

2 problems that deal with the root cause. 

3 Dealing with the root cause at the preventive 

4 level and fighting against time is really the incident 

5 management process. 

6 MR. GAMBALE: Gregg, I'll comment for DTCC on 

7 the front end of the question, which I think was along 

8 the lines of getting on top of and understanding when 

9 something is occurring. 

10 We use a lot of real time thresholds and 

11 alerts, state alerts that come to us. It is generally 

12 done on a risk based analysis of an application, so our 

13 clearing and settlement applications in particular throw 

14 a lot of alerts and state condition changes out to the 

15 staff, the interested staff. I'm an interested staff, so 

16 while I'm sitting up here on the panel, I'll ignore them. 

17 The point is that we are able to tell quickly 

18 when something is occurring, and frequently we can tell 

19 is it occurring internally, is it caused by something 

20 we're receiving from the industry. It lets us quickly 

21 adapt our incident management processes to address a 

22 problem. 

23 MS. EWING: At NASDAQ OMX, we run our own 

24 markets, and I'll talk about that framework. We also 

25 provide technology and in some cases run the operations 

0129

1 for other markets around the world through our Market 

2 Technology business. 

3 Our best practices have to be very clear, very 

4 documented, very predictable, because of the nature of 

5 the type of services we provide, in addition to the 

6 markets that we operate. 

7 Being a market operator, you are certainly 

8 concerned with your own particular issues, but you are 

9 also concerned about the other participants in the 

10 market, whether it's customer, another Exchange, et 

11 cetera. 

12 We have a lot of monitoring mechanisms in 

13 place, not just within technology. We have dedicated 

14 operations, and we touched upon that briefly this 

15 morning, the nature and knowledge and expertise that's 

16 needed to do that operation, and quite frankly, it's 

17 getting increasingly more complex. 

18 To deal with the complexity that's required and 

19 the monitoring that's involved, we are actually looking 
20 at more instrumentation, more self healing mechanisms, 

21 more technology, to help complement the oversight that 

22 takes place within our operation centers. 

23 In addition, we have our business operations 

24 functions as well as our market watch surveillance 

25 functions that are on a real time basis either monitoring 

0130

1 the health of the markets or reacting to issues that are 

2 taking place with our customers and other participants. 

3 Some of the very specific things we have done 

4 to protect the markets over the years, in particular, 

5 since the flash crash, are really bearing fruit in terms 

6 of helping to provide protection. If I take "clearly 

7 erroneous" as an example, we had a clearly erroneous 

8 function before, but we made the rules much more clear 

9 post-flash crash. That was one of the outcomes. 

10 Now when we look at our stat's, our clearly 

11 erroneous filing's have gone down over 60 percent in the 

12 past two years, just to give you a metric to think about. 

13 Monitoring, the nature of monitoring, who is 

14 doing the monitoring, how you are leveraging technology 

15 to augment the human interfaces, because there is a lot 

16 to monitor when you think about it. 

17 Best practices. Certainly, we use ITEL and 

18 best practices around that. 

19 The other thing, when you're in the line of 

20 fire, the more that can be pre-defined, the better. It's 

21 Crisis Management 101. 

22 The Code Blue process. You have a standing 

23 list of contacts, names, technologies on a different call 

24 than the business, outreach, outreach to the regulator, 

25 outreach, in our case, to the press many times. You need 

0131

1 to have a very structured escalation communication 

2 process in place. 

3 When you are also dealing with the problem, 

4 it's not always apparent immediately what the root cause 

5 is of that problem. The ability to detect, remediate, 

6 and bring the problem to resolution is really critical 

7 and how that is managed. We have again people on call, 

8 roles and responsibilities, all types of skill sets are 

9 needed. 

10 We talked a lot about software this morning, 

11 but our systems are more than software. It's the 

12 networks, the hardware, all the different mechanisms that 

13 connect it altogether, before you even leave your 

14 firewall. 

15 When you leave your firewall, you have all the 

16 interconnectedness with the other markets and with your 

17 customers. That in a nutshell, the technology, the 

18 monitoring, the process, the crisis management, really 

19 all come together in reaction to problems as they occur. 

20 MR. C. COOK: From our perspective, when we 

21 look at technology, risk, and the issues that come up, 
22 the interconnected aspect that you mentioned is really 

23 important. As you look across even just internal 

24 systems, there are so many moving parts that are 

25 connected, and with things that are working at such high 

0132

1 speed, it's really difficult often times to immediately 

2 point out where is that issue. 

3 Is it something on the internal end or 

4 something on the network, something at the Exchange side. 

5 From our perspective, the way that we build technology is 

6 with the very product focused aspect, more akin to our 

7 background in commercial product development, where we 

8 look at what we are building and we say what is it that 

9 people that are using this will expect out of it. 

10 It's much different to build an internal 

11 product where you have access to the code or you have 

12 immediate access to the networks, switches, routers, and 

13 things that you can immediately jump on and touch. 

14 We treat all of our internal operations folks 

15 as well as our clients as our own customers. 

16 When we build technology, we think of things in 

17 terms of not just what should the product do, but what 

18 shouldn't it do, and under the circumstances that we can 

19 enumerate, what are the situations and how do we want to 

20 allow people to respond to them in an effective way. 

21 For us, being in a production environment, 

22 obviously we're focused on getting people back to 

23 operation quickly, understanding what the ramifications 

24 of those issues are. 

25 I think a lot of it is driven by the technology 

0133

1 and the emphasis on instrumenting that technology to make 

2 it enabled to be managed and enabled to be controlled and 

3 monitored in such a way that when those events that we 

4 haven't planned for come up, everybody can quickly 

5 identify and have all the data they need in order to 

6 identify where that problem is. 

7 That is kind of an approach that we use on this 

8 stuff, along with all of the normal escalation procedures 

9 on the operational front. 

10 It is very much driven from a technology as a 

11 tool kind of mechanism. 

12 MR. BLOOM: I think all the panelists probably 

13 have similar -- I think someone said Crisis Management 

14 101 -- segregation of duties, this is what should happen 

15 in a crisis, segregate your people doing investigations 

16 from those who are figuring out what to do. 

17 When you are asking internal versus external, 

18 one of the biggest problems I've seen is when you don't

19 know. Sure, once I know if it's internal or external, 

20 it's a pretty easy play book, and usually I can figure it 

21 out and the team can figure out what it is we need to do. 

22 We try to make sure people in the role of 

23 triage, what is it our customers need from us in this 
24 situation, how can we, by plugging into all the trouble 

25 shooting and everything that is going on, figure out what 

0134

1 it is we can do to still be there for our customers while 

2 all this is going on, or do we have to just take 

3 ourselves out. 

4 I think that is a key point in the process that 

5 doesn't come out. Everyone has the documented incident 

6 but what are the trigger points, and when you don't know, 

7 how do you make those decisions and how do you sort of 

8 fulfill your obligations. 

9 MR. BERMAN: To ask a follow up question, Chad, 

10 you talked a little bit about product find, if I can make 

11 up that word, of how you think about the technology 

12 because you have had external clients before. 

13 That brings up an interesting point. During a 

14 monitoring crisis, at small firms and even at large 

15 firms, you need to get the head of development, the 

16 person who wrote the code, involved. They are the 

17 experts. That person may be on vacation. Maybe that 

18 person is not available. 

19 To what extent -- I think we asked a question 

20 this morning about separation of the quality assurance 

21 from the development process to maintain the independence 

22 -- to what extent do you think about the monitoring 

23 process as a process that needs to be done, certainly 

24 integrated, but independently, where you do not have to 

25 be the head of development in order to know what to do, 

0135

1 in order to diagnose a problem. 

2 MR. C. COOK: If I may, the productization of 

3 things is really important. When it comes to monitoring 

4 things, you have to put yourself in the mindset of who is 

5 going to be using it. 

6 Often times as engineers, it is very easy to 

7 just think of yourself as being highly technical and you 

8 have the ability to have a good understanding. 

9 It is important to kind of pull yourself back 

10 from that and say who are my constituents, who are the 

11 customers that are going to be using this, and what is 

12 their expectation and their level of expertise. That 

13 helps you tailor all of your functionality. 

14 When we build our technology, we are really 

15 instrumenting it not for us but more for the people that 

16 are going to be operating it on a daily basis, whether 

17 that is our customers or operational staff, those are the 

18 important people that we look towards. 

19 There is a feedback cycle where you work with 

20 them to say here is what we think your expectations are, 

21 here is what we are building for you, is this going to 

22 meet your needs, is it going to make it easy for you. 

23 That, to me, is one of the critical aspects of 

24 how you design these types of systems rather than just 

25 looking at okay, we have the technical staff on hand, 
0136

1 let's just escalate to the engineers where they can just 

2 jump right in. 

3 It makes it more efficient and it allows us to 

4 do more innovation and more development work if we build 

5 for those folks to self manage. 

6 MS. EWING: I would just like to add to that 

7 point. I couldn't agree with you more, Chad, in terms of 

8 productizing the operations aspect as much as possible. 

9 We actually have what we call a "design to 

10 operate" philosophy, where the operations team is 

11 actually part of the design and functional requirements 

12 phases. Sometimes an idea, even if you think about a 

13 business idea that may sound very interesting, may end up 

14 being really complicated to surveil, monitor, or operate. 

15 We actually even have input into the business 

16 concept that's being contemplated, and in addition, the 

17 instrumentation, the philosophy not just of how something 

18 is going to work, but how do you break it, and what 

19 happens when it's broken. 

20 You have some basic questions to just bring it 

21 to the mindset up front, and not just for technical 

22 issues. 

23 We as an SRO have obligations. We have rules 

24 that we file with the Commission that we have to adhere 

25 to. We also have a separate independent group to add to 

0137

1 the QA process that is actually part of our regulatory 

2 arm. 

3 This independent group does testing, not 

4 against the functional specification that is related to a 

5 system, they do testing against the rule book. It's a 

6 parallel segmented function that augments to again the 

7 ability -- in our case, it's not just a widget is broken, 

8 it's how we complied with the obligations and rules that 

9 we have to adhere to as a market. 

10 That's just as critical to us from a testing 

11 and monitoring perspective as latency and other types of 

12 issues. 

13 MR. STEINBERG: Gregg, just going back for a 

14 second to part of what I think was behind the question 

15 you asked, the notion that there might be only one 

16 developer to run to who understands the code when you're 

17 trying to de-bug, when you're trying to manage a 

18 crisis, the first thing we do is long before that, we 

19 establish a set of technology standards. 

20 By working with a pre-established set of 

21 standards, I know what my technology components are going 

22 to be, and I can afford to invest in training, level one, 

23 level two, and level three support staff on those 

24 technologies, so there isn't a single person I have to 

25 run back to. 

0138

1 At some point, we do go back to the developer, 
2 but we try very hard, and in fact, we have metrics around 

3 the percentage of incidents that are supposed to be 

4 resolved at each level without escalation to the next. 

5 The issue actually comes up in the period of 

6 time David mentioned where you just don't know what's 

7 going on, and you're still trying to triage it, so I 

8 wouldn't even know which experts to go get. 

9 At that point, it's the operational processes 

10 that are being discussed around quick triage. Then a 

11 quick decision about am I going to try to fix this or am 

12 I simply going to try to leverage what Dr. Markus said 

13 was the capacity to recover quickly, and not try to fix 

14 it, try to get things up and running while we get to the 

15 underlying defect afterwards. 

16 There is a very fast triage process we have to 

17 go to once we have decided what the technology component 

18 is. If we are going to fix it, if we are going to 

19 address it right now, then it's the notion of standards 

20 that allow us to have invested in the resources without 

21 having a single dependency. 

22 MR. JAHANI: I would like to take a step back. 

23 Please allow me to elaborate. When we develop software, 

24 when we build a system, the component of risk is always 

25 there. You have to look at how much risk can you 

0139

1 tolerate. 

2 In the financial industry, the simplest type of 

3 systems that we build for trading, only to have high risk 

4 systems, you need to do something like at least 15,000 to 

5 20,000 different types of test cases. 

6 When you go to medium risk, you have to do 

7 30,000 to 40,000 test cases. When you get to low risk, 

8 now you're talking about 300,000 to 400,000 different 

9 types of test cases. 

10 Please imagine when you're actually talking 

11 about testing, which we all want to do, it is absolutely 

12 impossible because it takes so much time. It's going to 

13 take so much time to just regress the systems, so it 

14 becomes basically impossible. 

15 In order to do so, the only way we can actually 

16 attack this crisis situation and so on is -- my apologies 

17 for the word -- "military precision." That's the only 

18 way we can really deal with our problems. 

19 When there is a problem, a crisis situation, it 

20 is no time to realize who is the best guy to be here, who 

21 is the best developer, who is the best QA guy, who is the 

22 best compliance person. 

23 First of all, everybody has to be hands on. 

24 That's the idea behind this Code Blue all hands on 

25 situation. 

0140

1 The second thing is we need to have rigorous 

2 military sort of disciplines in place so that we can 

3 absolutely do the right thing for the market, for our 
4 member firms, for the national market system. 

5 In this situation, there is really not much 

6 time to talk about, really, it's a fire drill, we are 

7 fighting against time. 

8 In order to address this situation, we 

9 absolutely have to start much earlier. We have to have 

10 mock testing, proper training. We have to make sure that 

11 there are instrumental testings as the back bone of this 

12 financial institution. 

13 We cannot operate the Exchanges and financial 

14 institutions no longer as a development shop. We have to 

15 do it as a production shop. That is when ITEL comes in. 

16 That is when proactive measures basically help us to do 

17 the right things. 

18 MR. R. COOK: Tom, did you have a question? 

19 MR. BAYER: I'm not signaling any group that 

20 has mentioned ITEL in the past, but Queen Elizabeth, who 

21 owns ITEL, I don't think ever thought about a case where 

22 a market would have a melt down or a problem like the 

23 flash crash. 

24 Knowing that IT people are introverted and they 

25 don't communicate well, we have talked about --

0141

1 MR. R. COOK: Tom, are you an IT person? 

2 (Laughter.) 

3 MR. BAYER: Yes. I don't communicate well in 

4 particular, but that's something else, from birth. 

5 What I think is important is once you have the 

6 crisis and you now have a problem and you're going 

7 through the ITEL functions to verify the systems are up 

8 and running and you're identifying root cause, et cetera, 

9 how do you handle the social aspects, i.e., the people 

10 that are calling the ball and the people that would do a 

11 kill switch or people that would handle the ultimate 

12 reaction, how are you handling the interaction and 

13 communication? 

14 You don't want to solve the problem while the 

15 patient -- if you were an emergency technician, you don't 

16 want the patient to die while you're administering help 

17 to the patient, you want to call 911 or you want to get 

18 additional help to come and solve the problem while 

19 you're working on the patient. 

20 How does that happen in the Exchanges and the 

21 self regulated organizations and how do you handle the 

22 communication and social aspects, how do you call each 

23 other, how do you bring them in, how do you bring each 

24 other up to speed on what the issues are and how do you 

25 resolve them, in general. 

0142

1 MS. EWING: I'll start. In our own markets as 

2 well, we provide, as I mentioned earlier, technology to 

3 other markets, so we have to have those mechanisms very 

4 highly defined and in place. I'm a big advocate of 

5 strong process, defined process. There are advocates of 
6 ITEL and other standards. At the end of the day, the 

7 process can't overrule the decision making and the common 

8 sense that's needed. So, the people matter. 

9 I've had the opportunity to work with the 

10 different markets, with very rigid structures, different 

11 degrees of rigidity, and in some cases, where everyone 

12 follows the right form, right process, they check the 

13 box, they told who they were supposed to, but you missed 

14 the bigger context, you missed the decision making, you 

15 have a longer term issue. 

16 It's really critical that in your crisis 

17 management defined process the people that are involved -

18 - as an Exchange, our first order of business beyond 

19 detecting what the issue is, is getting up and running or 

20 remediating the issue. Quite often we can tell what the 

21 root cause is right away. 

22 You need to have the culture that not just 

23 comes together and understands how to work in that crisis 

24 environment, but you also need a culture and an open 

25 communication and dialogue with those introverts that 

0143

1 maybe are the ones who oh, well, I did that change, maybe 

2 it has something to do with this, to have the ability and 

3 freedom to speak out without fear, without repercussion. 

4 That culture is an absolutely intrinsic part of 

5 a successful ability to work and fire fight in the type 

6 of environment that we operate in. 

7 MR. GAMBALE: I'd like to add one more piece to 

8 it. I agree with Anna completely. 

9 I would point out to also address the point of 

10 the lone developer or sole developer, one thing that we 

11 do at DTCC, we have frequent exercises operating out of 

12 multiple locations, we will stand a location down for the 

13 day, and we will announce that the New York location will 

14 not handle any issues that come up. We can override that 

15 if we had to. 

16 The routine crisis and instance that occur are 

17 handled by an alternate location, a location in Tampa or 

18 location at other sites in the United States. 

19 We will rotate the responsibility, so on these 

20 stand down days, it's a little bit of a holiday, if you 

21 will, from crisis, but it lets the staff -- it gives a 

22 broader sense that the staff all around the United States 

23 can handle a situation because they're flying on their 

24 own pretty much on most days. 

25 MR. C. COOK: If I may, on the engineering 

0144

1 personality aspect of it, we are heavily an engineering 

2 shop. I understand what you mean on many levels. 

3 I think from that perspective, 

4 it's really a focal point of how we build and what we are 

5 building for, so really when you look at the customers 

6 and when there's an issue that comes up, our goal is we 

7 want to get the customer back in operation as quickly as 
8 possible. 

9 It's less about who messed up, who made a bug, 

10 who had some issue, who pulled the wrong plug, and more 

11 about how do we work together to fix this and make the 

12 person or people or clients or whatever back in business 

13 in a meaningful way and very quickly, and with that 

14 obviously, all the right technology around it to support 

15 in that effort. 

16 There is a certain transparency that we build 

17 into our culture. I agree with Anna's points. The 

18 culture itself of engineering is very important. This 

19 carries through to how you look at transparency within 

20 products. The previous panel, for those that saw it, 

21 they were talking a lot about you can't prevent 

22 technology issues. Really, it's about how do you recover 

23 from them. 

24 In that, it's not about pointing fingers, it's 

25 really about making sure that people know bugs happen. 

0145

1 We don't want them to happen. We are going to put all 

2 the rigorous testing and controls in place that we can, 

3 but it's still not going to solve every issue. 

4 Instead, let's focus on being effective and 

5 proactive in how we monitor, how we can identify where 

6 the issue is so we can isolate it with the focal point of 

7 getting them back in business. 

8 MR. BLOOM: When we are hiring people, the 

9 engineering discipline, those who are great at building 

10 your system and figuring out root cause after something 

11 happens, in my experience, they are usually the least 

12 well suited to do the triage while it's happening because 

13 they are too intrigued by what is actually happening 

14 compared to what do I need to do in the marketplace. 

15 We have segregated operations staff where we 

16 try to hire what we call "applied engineers" as compared 

17 to the ones who want to be in the middle of it, and let 

18 both people do what they do best, and have a manager in 

19 charge who will make sure the right investigations are 

20 happening. 

21 It follows up on your question about the 

22 original developer. That's for the ultra emergency, 

23 figure out the root cause, but typically in the heat of 

24 what's happening, that's not the person we want to go to.

25 MR. BAYER: Is the incident manager also the 

0146

1 person who communicates with the senior management team 

2 and keeps them informed, and is that uniform across the 

3 panel? 

4 MR. BLOOM: For us, that's the case. 

5 MS. EWING: It varies depending on the 

6 incident. We have different levels of code. We have 

7 Code Blue, which is crisis. We have everyone in the 

8 organization from all the different areas. 

9 If there is senior executive management on a 
10 Code Blue level, the decision making is made at that 

11 level, because you have regulatory implications and other 

12 things to think about beyond the technical. 

13 We have incidents every day. We all do. There 

14 are different types of incidents. It could be a router. 

15 It could be a hardware failure, et cetera. 

16 We have different levels. That is more the 

17 incident manager tends to be the one that runs with the 

18 full issue. 

19 MR. GAMBALE: If it's a big enough incident, we 

20 will run with two incident managers. One will be the 

21 business and regulatory liaison person working with the 

22 executives, the business, the regulators, and there is a 

23 technical incident manager who is working on technical 

24 with the engineers and developers. 

25 MR. JAHANI: I just want to agree with 

0147

1 everything that was said so far. I want to say we have 

2 to distinguish the problem management from incident 

3 management. Incident management really is focused on 

4 immediate recovery and basically mitigating the problems 

5 so we don't have an impact on the member firms and the 

6 market. The incident manager is literally the man in 

7 charge in that situation. If there is a crisis, the 

8 incident manager is in charge. 

9 However, there is also an incident management 

10 group which basically is a broader group that takes the 

11 incident after recovery and basically passes it on to the 

12 problem management process, making sure that the problem 

13 gets basically resolved, the root cause gets addressed. 

14 In other words, if we don't act this way, and 

15 we kind of categorize the incidents from say low risk 

16 incident to high risk incident, then we are practically 

17 killing the whole idea. 

18 The incident manager has to be the person that 

19 is in charge. That doesn't take away the accountability 

20 that goes with CIOs, CTOs, or a chief operating officer. 

21 They are also part of the problem -- excuse me -- the 

22 problem management groups as such. 

23 (Laughter.) 

24 MR. JAHANI: The COO of the company, the CIO of 

25 the company, head of Compliance, they all have to be part 

0148

1 of the incident management group and work on this 

2 together. 

3 MR. BURNS: This is a very good dialogue about 

4 internal management and communication. What about 

5 communication externally, both to us as regulators and to 

6 your customers, market participants, who are wondering 

7 what's happening over there? 

8 MR. JAHANI: In our case, like I said, as soon 

9 as there is a problem, the first person that is actually 

10 there and witnessing everything is the compliance 

11 officer. 
12 As soon as we have an incident, a person from 

13 the Compliance group has to be there. During all our 

14 operational hours, we have full coverage from the 

15 Compliance group. They are documenting everything. 

16 When the incident response basically is ready, 

17 that's when it formally gets communicated to the SEC, but 

18 while the problem is ongoing, when the incident is 

19 basically under management, in this situation, we 

20 immediately contact the SEC and let them know about the 

21 problem, so they are actually expecting a response as 

22 soon as possible. 

23 MR. STEINBERG: In our case, the process is 

24 managed by one group consistently, the people that do the 

25 communications both internally and externally. Change is 

0149

1 a function of the incident, not just the type of incident 

2 but the instant materiality. 

3 Every single incident that we look at gets what 

4 we call a "materiality score" associated with it, which 

5 is a function of how many clients were affected, how long 

6 has it been happening or how long were they affected for, 

7 how critical are the systems that are involved, and what 

8 was the time of day, so open market, closed, it is very 

9 different from after hours. Based on that, different 

10 groups will have different responsibilities to 

11 communicate both internally and externally, senior 

12 management, regulators, et cetera. 

13 MS. EWING: I would just add we have different 

14 mechanisms for communication. We have the same roles and 

15 responsibilities around the regulatory outreach. We have 

16 a group that does the outreach to the regulators. The 

17 client desk, the business operations function does 

18 outreach to customers. 

19 We send out system status alerts when we even 

20 detect that we may be having an issue. That is something 

21 we will do even before we confirm there is an actual 

22 issue. The more outreach and communication you can do, 

23 the better, and the more real time you can do it, the 

24 better. 

25 If there are crisis situations or if we have 

0150

1 big known events, like a Russell re-balance, et cetera, 

2 we will set up industry-wide calls, and not just with 

3 ourselves, but with the other Exchanges. Those will be 

4 calls in place that are pre-planned that will take place. 

5 MR. C. COOK: I'd like to offer a slightly 

6 different perspective. We are a pretty small 

7 organization. We don't have the large teams of staff 

8 that are dedicated to doing one function or the other. 

9 What we have built into our culture, and this 

10 is probably pertinent to those folks out there that do 

11 have smaller staffs that are in a very operational 

12 capacity, we build it into our culture that part of that 

13 job even as an engineer is an operational aspect. 
14 We find it to be very valuable. As I mentioned 

15 earlier, there is that feedback loop between what we 

16 would call our customer or constituent and how you evolve 

17 that technology in a way that works for them. 

18 Having our engineers very much engaged by 

19 matter of course because we don't have that many people, 

20 having them actively engaged in any incident that comes 

21 up helps provide that feedback mechanism that we then use 

22 to improve how we do things. 

23 We are focused a lot on automation and enabling 

24 our constituents to function on their own, self 

25 management, and things of that nature. 

0151

1 That feedback loop is sort of built into how we 

2 operate. It's a little bit different than a large 

3 organization. 

4 MR. R. COOK: Moving on to a slightly different 

5 topic, talk to us a little bit about kill switches. 

6 There has been a lot of discussion of this in the press, 

7 a lot of interest. 

8 Help us unpack this concept a little bit, maybe 

9 peel back the layers of the onion a little bit and help 

10 us understand what are the different types of kill 

11 switches, how do you think about designing them, how do 

12 you think about what triggers them, how do you think 

13 about for those that require some human intervention, 

14 when is that the right way to go, who makes that 

15 decision. 

16 What kind of process do you have around it, 

17 just sort of broadly help educate us around that whole 

18 concept. 

19 MR. STEINBERG: I'll bite. 

20 (Laughter.) 

21 MR. STEINBERG: There are a couple of things to 

22 remember about kill switches. In our view, it's a kill 

23 switch if somebody does it to you, and it's a suicide 

24 switch if you do it to yourself. There is a big, big 

25 difference. 

0152

1 People are going to be reluctant to 

2 systemically cut themselves off from the market. Any 

3 attempt to sort of pre-define automated thresholds that 

4 will in an automated way disconnect you feels like a 

5 suicide switch. The thresholds are going to be set very 

6 conservatively to a point where they may not actually 

7 operate when you want them to, at least not in the way 

8 you want them to. The counter to that is complex systems 

9 that are failing, can't be relied upon to manage 

10 themselves through that failure. 

11 Again, we're going to worry a lot about 

12 automation kicking in at the wrong time, and perhaps 

13 de-stabilizing a system that shouldn't be de-stabilized. 

14 The ability to detect unusual behavior, whether it's 

15 based on volume or one of the many different parameters, 
16 it is incredibly important to us, it's incredibly 

17 important throughout the whole system, when unusual 

18 behavior is detected, we have the ability to do select 

19 kills, which is to route away maybe to a different 

20 market maker. 

21 It would probably make sense for the market 

22 makers and the Exchanges to have the ability to detect 

23 unusual behavior and reach out to us and ask why, and 

24 again, back to the materiality. We either have a fairly 

25 short amount of time to answer the question why is this 

0153

1 unusual behavior happening or a little more time if it's 

2 less material. Answering the question and saying no, 

3 this is actually expected -- I'll give you an example. 

4 If we see unusual behavior with one market 

5 maker, we may choose to route away and route to a 

6 different one. That's probably in the best interest of 

7 our clients and the market. If that other market maker 

8 were to suddenly see a spike of activity from us because 

9 we are intentionally routing to them, we wouldn't want 

10 them to activate an automated kill switch and shut us off 

11 because that would in fact de-stabilize an environment 

12 that we are trying to add stability into. 

13 A quick conversation about if we can't control 

14 it at the micro segment level, having an Exchange or 

15 market maker come back to us and say based on the 

16 materiality, based on the level of impact that we see, 

17 your impact comment, we will give you five minutes to 

18 explain to us if this is normal or if this is unusual, or 

19 we will give you ten minutes, and then we're going to cut 

20 you off if you can't. If you can convince us, then 

21 that's fine. 

22 We see sort of a layered approach to these 

23 things that probably makes the most sense with a human 

24 discussion tied to it. 

25 MR. BURNS: Just one question. Maybe it 

0154

1 doesn't translate but in the "clearly erroneous" context, 

2 we once had sort of the discretion versus very clear 

3 parameters and rules by which everyone has operated, so 

4 is this different in the kill switch environment? 

5 If it is sort of a discretionary or ad hoc,

6 what's going on over there, Fred, kind of situation, when 

7 no one else might be in that loop, does that create the 

8 uncertainty that drove away liquidity on May 6? 

9 MR. STEINBERG: I'm not sure it does. First of 

10 all, it doesn't have to be discretionary in terms of 

11 triggering the conversation. That can be pre-defined and 

12 that can be well defined. That's the first thing I would 

13 say. 

14 The second thing is that there is some 

15 discretion about is this expected, is this normal, can 

16 you explain it to me. If we can to your satisfaction, 

17 "you" being the Exchange, you being the market maker, 
18 that's fine. 

19 It is also not self said. The check and 

20 balance here is other than establishing the initial set 

21 of thresholds that would trigger that conversation, we're 

22 not doing "clearly erroneous" to ourselves, somebody else 

23 is saying hang on, now you have to convince me this is 

24 right, so it's not a loosey-goosey we're just checking 

25 ourselves and it looks okay. 

0155

1 MS. EWING: Just to respond to the "clearly 

2 erroneous" example and the stat I gave earlier, it is 

3 important for in this case the Exchange and our customers 

4 to have a very clear framework of how decisions are made. 

5 It has to be as black and white as possible. 

6 When we went with the "clearly erroneous" and had very 

7 specific thresholds established for the different price 

8 points, et cetera, that provided a lot more clarity and 

9 facilitated that process, and certainly limit up and 

10 limit down will continue to advance that capability. 

11 Kill switches need to happen at multiple 

12 layers. The working group we are a part of and we are 

13 very much a proponent of the concept of the kill switch, 

14 and at the Exchange level, it needs to complement other 

15 risk controls that are already in place, needs to 

16 complement what's in place for obligation to the market 

17 access rules at the broker-dealer level. 

18 I think kill switches are important, but we 

19 need to ensure we don't think of them as the big red easy 

20 button. It's layered. It's complex. There is decision 

21 making criteria. There is that human element involved. 

22 For example, one of the things we are thinking 

23 about is if you reached a certain threshold, regardless 

24 of what you end up using, we're talking about the peak 

25 net notional right now as the metric, you do an outreach, 

0156

1 you make a phone call. You are at this level, is there 

2 something wrong, is this what you intended, do we need to 

3 increase your limits. 

4 You definitely need that human dialogue to take 

5 place, but again, it has to be against some very defined 

6 set of guidelines and metrics. 

7 CHAIRMAN SCHAPIRO: Is there time for that? It 

8 doesn't take very long in these markets, a couple of 

9 minutes, for an enormous amount of damage to be done. If 

10 a kill switch isn't automatic, there are pre-determined 

11 triggers and that shuts the trading down, but there's a 

12 conversation, are we checking ourselves, you have a lot 

13 of problems along the way, just from taking that time to 

14 do that. 

15 MS. EWING: I think that is a challenge. I 

16 think that is one of the areas we are talking about and 

17 trying to define. 

18 It depends on where we put the threshold. 

19 There is a point in time when we make the outreach. At 
20 100 percent, if I can use an analogy, let's say at 70 

21 percent, we make the outreach. If it hits 100 percent, 

22 the kill switch would probably have to trigger. 

23 Those are some of the details we are talking 

24 about and designing them as we speak, and it's all about 

25 the unintended consequences of some of those decisions. 

0157

1 It could be a very high volatility day, and 

2 that's natural order flow that we are dealing with, as an 

3 example. 

4 Those are some of the considerations we have to 

5 design into the model itself. Again, we feel very 

6 strongly it has to be a layered approach. The kill 

7 switch at the Exchange level is really the last resort. 

8 MR. C. COOK: On the kill switch side, I think, 

9 I agree with your points. It's almost like we're on the 

10 same team here. 

11 I think of it in terms of the security world, a 

12 defense in depth kind of approach. Everybody has a 

13 different context of how they're looking at things, their 

14 angle, what their risk limits are, what their liabilities 

15 are. 

16 The broker, the clearing firm, the DTCC, the 

17 Exchange, everybody is looking at one piece of the puzzle 

18 or several pieces of the puzzle, and I think 

19 collectively, you can arrive at the whole picture by 

20 putting them altogether, but that is obviously a pretty 

21 onerous task. 

22 I guess from one perspective, the kill switches 

23 are almost going to need to exist at multiple locations 

24 and multiple places, but there has to be some form of 

25 normalized interaction where we can make that 

0158

1 communication mechanism more efficient, and whether 

2 that's a protocol, a process, to talk with one another 

3 and say we have identified a problem on our end, do you 

4 see anything that would lead to a problem on your end. 

5 The ramifications of some of these issues now, 

6 especially at the speeds at which we're operating, can go 

7 from very minor to very significant very, very quickly. 

8 I think the layered approach absolutely makes 

9 sense, but then it's how do we tie those layers together 

10 and how can we work together as a community to formalize 

11 those relationships. 

12 Typically, it's very difficult to do that with 

13 some firms because they are so proprietary in nature and 

14 there is a lot of special sauce and things that everybody 

15 has talked about. 

16 I think outside the scope of the sauce, we 

17 don't need to interfere with what their techniques are, 

18 but we can at least arrive at a common view of risk and 

19 how to react to it, or at least give some sense of 

20 notification when we see issues. 

21 MR. GAMBALE: Just to comment on the 
22 coordination needed from the different trade sources for 

23 equities. When I came to DTCC 15 years ago, the morning 

24 report that showed me my trades that came in overnight 

25 and the day showed me half a dozen trade sources. Now, 

0159

1 it has 50 equity trade sources. 

2 I think there is a lot of coordination that 

3 needs to be done on a kill switch across all the various 

4 sources. 

5 MR. R. COOK: I know there is a working group 

6 that is talking about that. What are some of the ideas 

7 that are at least floating around without necessarily 

8 needing to endorse any of them about how you deal with 

9 the fact that there is not just one Exchange, there are 

10 multiple Exchanges. 

11 For example, if a threshold is hit at one 

12 Exchange, what kind of communication have people been 

13 talking about to other market participants for whom that 

14 might be relevant, and what are some of the concerns 

15 people have around any of that kind of coordination. 

16 MR. JAHANI: I would like to comment on that. 

17 I think everybody generally speaking agrees kill switches 

18 are good and they should be implemented somehow. The 

19 question is whether or not we want to implement a red 

20 light or stop sign. 

21 In order to be more specific on this, I think 

22 what we are really trying to say is we have to define the 

23 situation, the scenarios really, where kill switches 

24 should apply. If so, the clear protocol has to be 

25 properly identified, the thresholds have to be clearly 

0160

1 defined and so on. I think if we can actually create 

2 that, it is a viable solution, and it should be used. 

3 The human component, as mentioned here, it's 

4 very critical because we just don't want to do the wrong 

5 things, so if you're automating, obviously some of the 

6 scenarios definitely can be automated. The question is 

7 how many. 

8 I think there has to be a work group that 

9 really works on the details and comes up with clear 

10 scenarios and clear protocols and procedures. 

11 MR. BAYER: Do you envision that to be the 

12 equivalent of an open source kind of consortium where you 

13 would share monitoring activities, tools, and other best 

14 practices, and you could share them across your quality 

15 assurance staff, for example, so you would have an open 

16 source library that all the Exchanges and market 

17 participants could contribute? 

18 Have you ever thought about that, for example? 

19 MR. JAHANI: I believe this problem has to be 

20 taken back to its infancy. In other words, how we design 

21 systems. Please bear in mind many of our applications 

22 are not even properly monitorable because when we 

23 actually are developing code, in accordance with our 
24 system development life cycles, we try to make them as 

25 efficient as possible, as fast as possible. 

0161

1 In order to really be able to get the right 

2 picture, in order to get the right flag, this is 

3 something happening, right, we need to start coding 

4 differently. 

5 In order to do so, we have to practically look 

6 at the entire ecosystem. 

7 I don't think any of our institutions nowadays 

8 should act as one single entity. We all are part of the 

9 same ecosystem and we have to see this as our ecosystem. 

10 Actually, we have to even take it back to our vendors, 

11 to our partners. 

12 This is not any more a financial industry 

13 problem. People that sell equipment to us, people that 

14 sell code to us, everybody is part of the same ecosystem. 

15 We have to take it back to the design level and start 

16 doing things a little differently. 

17 I think those scenarios all have to be 

18 identified and worked on. I definitely would agree it 

19 has to be a consortium of different roles and 

20 responsibilities that are incorporated there. 

21 MS. EWING: I'd like to talk about one area 

22 that was actually talked about quite a bit this morning 

23 because I think it is very relevant here. 

24 That is in the area of testing. We talked 

25 about the interconnectedness and ability to do our own 

0162

1 unit testing, and we have automated test suites and tools 

2 that we use to do functional testing, non-functional 

3 testing, latency, et cetera. Again, going back to 

4 technology as part of the solution and tools are in 

5 place. 

6 Where the complexity in testing really is 

7 involved is the fact that we have multiple markets that

8 are interconnected, multiple participants, multiple 

9 protocols that connect into these markets. 

10 We were born as an electronic market, so in our 

11 DNA, to be an open system and then how do you test 

12 towards being an open system. We have test symbols. We 

13 have had test symbols for at least ten years, maybe 

14 longer, where firms can come in, in production, and test 

15 their applications or test our capabilities, 

16 recertifying new companies as they join our markets. 

17 We have a dedicated test facility that emulates 

18 production, in addition to being able to test in our 

19 production environment. We do weekend tests. 

20 As an industry, and again, I've been with 

21 NASDAQ for 12 years and have been very much involved with 

22 a lot of industry weekend type testing that we used to 

23 do, that I'm going to stay one step short of mandatory, 

24 but firms would come in on weekends, and we would test. 

25 In those days, it was more new functionality or migration 
0163

1 to a new platform, et cetera. 

2 I think one of the things that we can and 

3 should do a lot more of as an industry is system-wide 

4 testing. System-wide testing that doesn't just test can 

5 we ping our DR site, right. Can we connect to an 

6 Exchange. We have to look at how we can test a brake, 

7 how we can test a brake as an industry, all these things 

8 we are talking about, including kill switch scenarios. 

9 I think that is a working group, we have 

10 different tracks. One of the tracks we have is to look 

11 at best practices around not just the development cycle 

12 but the testing itself. We are a big advocate of that. 

13 We have been big investors in test symbols, in production 

14 test environments, and we would advocate and highly 

15 participate and coordinate with the industry on these 

16 types of tests. 

17 I think that's the maturity level we are at 

18 now, and we need to do more of that. That's the only way 

19 when you have your mindset about how do you brake the 

20 thing, that's really where we need to focus next, as part 

21 of your ecosystem question. 

22 COMMISSIONER WALTER: Can I ask a question to 

23 pick up on that and to bring in both this morning's 

24 discussion, some of which I had to skip, unfortunately, 

25 and this afternoon's. 

0164

1 Given the kind of interconnectedness that 

2 exists today, both in terms of systems and perhaps even 

3 more importantly in terms of reputation and the 

4 reputation of the market function, how do you assure 

5 everybody plays at the same level? 

6 We are all familiar with situations when 

7 confronted with should I spend my million dollars putting 

8 extra controls into my system or should I spend it on 

9 something that may bring profits in the door a little 

10 faster. Not everyone draws what we would call the right 

11 conclusion to those kinds of questions. 

12 Is that the place for oversight that Professor 

13 Leveson talked about this morning? How do you make sure 

14 everybody is in the room? If you hold the test and some 

15 major player doesn't come, you have a hole in that 

16 system. How do you get to it's really going to be done? 

17 MR. STEINBERG: In a complex interconnected 

18 system, there are very often key points where there is a 

19 lot of information flowing through. You can call it a 

20 "choke point" or a "concentration point." 

21 Those are the places where you get the most 

22 efficient oversight. If you try to distribute it out to 

23 the edge, you are absolutely right, people tend to opt 

24 out or interpret differently. 

25 As people come together at the market makers or 

0165

1 at the Exchanges, you get an opportunity to start to 
2 build the layered approach to controls and governance 

3 that is exactly what you're talking about. 

4 You also lose something at the same time, which 

5 is you begin to extract what's happening at the edge, so 

6 with testing in particular, there is this notion of what 

7 the QA folks call "black box testing" versus "white box 

8 testing." 

9 Black box says I know nothing about what's 

10 inside, I can load it, I can test it, I can see if it 

11 behaves to stimulus the way I expect it to, but I don't 

12 know what it's really doing. 

13 White box testing says I deeply understand 

14 what's inside, and I can exercise very specific 

15 scenarios. To do white box testing, you kind of have to 

16 do that at the edge and you have to enforce that at the 

17 edge. 

18 I'm not sure you can concentrate that in an 

19 industry-wide event simply because everybody's inside is 

20 different. Each participant is going to know how their 

21 system should behave on the inside. 

22 It's both. It's a set of standards for 

23 individual participants to do things like white box 

24 testing, load and limit, test to fail, scenario planning, 

25 that kind of stuff, and then at the concentration points 

0166

1 of the interconnected system, like the Exchanges, it's 

2 additional oversight, exactly what you are talking about. 

3 It's the combination of the two, I believe. 

4 COMMISSIONER WALTER: At some level, doesn't it 

5 require some sort of regulatory mandate, not at the micro 

6 level, but in a more principle based kind of way? 

7 Otherwise, you end up with somebody opting out. 

8 MR. STEINBERG: Yes. In a principle based kind 

9 of way, yes. The challenge would be if you tried to 

10 define at the micro level the activities at sort of the 

11 black box level, you would have an mismatch between --

12 COMMISSIONER WALTER: It has to be both 

13 principles based and enforceable. 

14 MR. STEINBERG: Of course. 

15 MS. EWING: We touched upon this this morning 

16 and we talked about the ARP and the ARP program. I 

17 actually forgot it was voluntary because we take it very 

18 seriously. That is again a program. 

19 When you couple what you do with ARP, and we 

20 didn't touch upon this, this needs its own roundtable, 

21 cyber security, when you think of everything we are doing 

22 in the cyber security front, there is actually a lot of 

23 synergy and interplay between those types of programs. 

24 I think we have some building blocks, some 

25 audits. We have a very strong internal audit function as 

0167

1 well as we use external auditors to really test against 

2 best practices and standards. 

3 Standards exist. We don't have to invent them. 
4 We don't have to create them. We need to maybe choose 

5 them or have a short list and adopt them. We don't need 

6 to create necessarily new rules or new sets of standards. 

7 Again, I think the oversight through the ARP program is 

8 something that I think can be expanded. 

9 MR. JAHANI: I'd like to say a few words about 

10 this. One thing that we all have to remember is when we 

11 are introducing new functionality, at least 99 percent 

12 gets tested internally in-house. All the Exchanges, all 

13 the financial institutions, they all want to do the right 

14 things. In other words, they do a lot of testing. They 

15 do as much testing as they think they have to. 

16 The problem is basically when the issue gets 

17 out of control. In other words, it gets outside the four 

18 walls. This is when it becomes what we have experienced 

19 the last few years. 

20 I think it was actually mentioned this morning 

21 by Chris Isaacson. Many of the Exchanges today basically 

22 have a Gemini environment. We all basically subscribe to 

23 the same market data, same production flow/order flow 

24 that comes in and so forth, and we constantly test that 

25 internally for our own use. 

0168

1 My feeling is there has to be some sort of 

2 interconnected way. We have to kind of mandate the 

3 testing environment not should be used only on weekends, 

4 it should be used in parallel with our production, so 

5 that all the new type of orders that are coming in -- I 

6 know people see dollars and costs -- the fact of the 

7 matter is in the long run, this is going to become a 

8 major savings. 

9 We are already operating in these environments. 

10 What if we connect them to each other, what if we 

11 actually test the test symbols or basically the real time 

12 order flow that we get, and we actually communicate to 

13 each other by facts? In other words, this is our execute 

14 on your new type of order. What is it you see here and 

15 so on. 

16 That's when we actually start seeing the 

17 impact, and that's when we can actually report back to 

18 each other and help improve. 

19 MR. BERMAN: I'd like to come back to the --

20 sorry, Tom. 

21 MR. BAYER: Just real quick, one comment that I 

22 would add to the concept we're talking about. Anna, your 

23 concept of weekend testing, we could potentially start 

24 with a generic test period, then you could layer in some 

25 of the white box scenarios in a second test run, and 

0169

1 then in a third test run, we could potentially layer in 

2 cyber security. 

3 If it was all automated, we could then 

4 understand the test results. That may reduce the overall 

5 costs. 
6 MS. EWING: Again, going back as a veteran who 

7 used to do a lot of weekend testing, again, it wasn't 

8 mandatory but firms had to show up because we had major 

9 changes going on. 

10 What has happened is there are just so many 

11 venues and so many things to test. It is about creating 

12 not necessarily more work or more effort for the 

13 industry, it is hopefully doing it in a way that can 

14 streamline and collaborate and coordinate amongst each 

15 other so that we can do the type of testing that's 

16 needed. We have an industry-wide test coming up in a few 

17 weeks to look at our DR resiliency type testing. 

18 We can do it as an industry. We know how to do 

19 it. We can be as efficient as possible. I think we can 

20 have different themes, and this is just brain storming 

21 now, different themes around what we focus on in any one 

22 particular test, so firms can come to the table and test. 

23 Their internal systems, I absolutely get. 

24 We're not pretending to test beyond our firewalls, but we 

25 want to provide the infrastructure and the environment, 

0170

1 and with ourselves, with our data vendors and other third 

2 parties, who need to also participate and be a part of 

3 that testing. 

4 MR. BERMAN: I don't want to let you guys off 

5 that easy on the kill switch question. Just based on the 

6 initial comments to Robert's question on the kill switch, 

7 lots of complexities. There may be room for additional 

8 human layers, and there are certainly some pro's and 

9 con's. There is a very concrete idea that I think the 

10 working group has put out, and it's pretty 

11 straightforward, based on just the comment letter. 

12 At each market center, yes, things are 

13 interlinked, but at each individual market center, there 

14 would be a programmatic way of the market center to say 

15 you can't trade, you have gone beyond the limit and that 

16 would be automated. 

17 There might be a human call that goes out, but 

18 if there's no time for that call, it gets automated. If 

19 I'm calling you on the phone and saying you just hit your 

20 100 percent limit, sorry about that, you're off, that's a 

21 very concrete idea that was proposed. 

22 I just want to drill down into that and get 

23 some opinions. I think in Ameritrade's comment letter, 

24 Lou, you were specific. I think you said no, you would 

25 prefer there not to be that level of automation. 

0171

1 I think in some of the other comment letters it 

2 was yes, we understand the complexity of that, and we 

3 understand that sometimes you might be shut off and it 

4 might make things worse, but on balance, yes, we do want 

5 that. 

6 I would just like to drill down a bit into 

7 that. I think that is going to be an idea that is going 
8 to gain a lot of interest and a lot of traction over the 

9 upcoming months. 

10 MR. C. COOK: If I may, tying in the two topics 

11 of testing and the kill switch, the thing I always worry 

12 about when it comes to testing is the gap. What piece of 

13 the infrastructure, the environment or whatever, that 

14 can't be accounted for when you are doing all these tests 

15 even under the many tens of hundreds of thousands of 

16 cases. 

17 To me, it comes down to there is one aspect of 

18 it which is just pure control. You can never control the 

19 operating systems you are using. You can never control 

20 the gear you're using. All these third party vendors and 

21 other libraries and things like that. At some basic 

22 level, you need almost that proverbial knife switch where 

23 you can pull the plug. 

24 At that level, that obviously has major 

25 ramifications. I think as part of this kill switch idea, 

0172

1 one of the most important aspects of understanding, going 

2 back again to sort of the risk management view of the 

3 world, looking at it from a technology perspective and 

4 saying what are the ramifications when we do pull that 

5 switch and cut off the flow of trading, what are the 

6 results, and what are those issues and the downside risks 

7 to the people we just cut off, and how do we work around 

8 those things. 

9 I think we need to form a comprehensive view of 

10 what we want the kill switch to do, but also know what 

11 the results could be so that we can build into it other 

12 types of technology that can help offset some of the 

13 downside issues. 

14 You had mentioned the suicide switch and that 

15 kind of thing. It's because you have those different 

16 perspectives of the person that is affected or the one 

17 that is actually pulling the switch to save themselves. 

18 I think there is a lot of room in there for development 

19 to figure out how do we solve some of those 

20 ramifications. 

21 MS. EWING: I think we believe that we can 

22 implement a kill switch solution and we are advocating as 

23 part of the working group that we work towards that goal. 

24 We probably got into a little bit of a 

25 requirements definition mode a few minutes ago because 

0173

1 there are a lot of details to think about in terms of how 

2 it is designed, what kind of monitoring is needed, what 

3 kind of alerts, again, beyond the human outreach, from an 

4 automated perspective, what kind of measures do we need 

5 to put in place. 

6 We have technology. Firms use risk management 

7 technology. We are a provider through F-10 that is used 

8 by broker-dealers to not just do their risk controls 

9 against a particular market or even particular asset 
10 class, but across asset classes. 

11 When you look at some of the controls that we 

12 are thinking about, whether it's credit, capacity type 

13 volumes that we want to think about, it goes beyond an 

14 one to one ratio, one to one venue. When we say it's a 

15 layered approach, it is because we believe it needs to 

16 be. 

17 Having said that as an Exchange, we are working 

18 with defining the mechanisms of what the rules would be, 

19 how we calculate it, we have this peak notional value 

20 that we are talking about, but what kind of monitoring 

21 and alerts also need to be built as part of this. 

22 That is another key part of what needs to be 

23 built as part of the solution, that firms can get real 

24 time messages back from the Exchanges of a pattern. This 

25 is artificial intelligence to some degree, starting to 

0174

1 cull some of the trends that we are seeing, et cetera. 

2 There are a whole bunch of things we are 

3 talking about that we can augment to the on/off switch, 

4 but those are the things I think need to be part of the 

5 consideration. Again, we are supporting the concept and 

6 advocating a solution. We are just driving to the 

7 details through our working group. 

8 MR. BLOOM: Our firm is in support of that 

9 concept as well. Our view is at the start, everyone is 

10 talking about testing procedures and what not. The 

11 automated kill switches need to go through that same 

12 level of rigor. 

13 One of the things we have to be careful about 

14 when we are setting those thresholds, be it at 75 percent 

15 where you get the phone call, versus 100 percent, where 

16 you're done, that we set those levels intelligently. 

17 I think that is where a lot of the debate is going 

18 to come from because the last thing we want to see is a 

19 well intended kill switch disrupting proper market 

20 activity, the failure case there. 

21 We have a lot of kill switches we run 

22 internally. I guess kill switches are now the trend, 

23 almost as credit limits. Customers come into us too 

24 often, guess what, you're done. We're not trading any 

25 more. This whole layered thing where if we see too much 

0175

1 activity coming from one customer, they are tapped out. 

2 If we see irregular activity on a particular 

3 venue, any of our operational staff can say we're going 

4 to disconnect ourselves from that venue and our smart 

5 order routers will take over from there. 

6 What we are talking about here is an industry 

7 internal thing, another fail safe, guess what, maybe we 

8 are the ones doing it, who knows, we hope not, yet 

9 something sees on the Exchange irregular activity, you 

10 get the phone call first, if you don't respond fast 

11 enough and you have blown through the second limit, in 
12 the interest of market safety, we're supportive of that 

13 automated thing happening, but again, we have to work our 

14 way into the automation using all the testing principles 

15 we talked about earlier so we don't disrupt valid market 

16 activity in the interest of trying to be more aggressive. 

17 MR. STEINBERG: My only comment would be first, 

18 we are supportive of the concept of automated monitoring, 

19 automated alerting. We certainly have the same view of 

20 layers of kill switches. We run lots of rules starting 

21 with each order. Does the client have the cash. Is this 

22 position kind of unusual, et cetera. We start at the 

23 individual transaction level and then build up. 

24 What we are talking about now is a much more 

25 aggregated, much more concentrated at a place in the 

0176

1 system where we have a lot more visibility into like a 

2 market, how the whole system is behaving, so it makes 

3 sense to add additional controls there. 

4 Our point of view is simply that getting the 

5 phone call first is a great idea, depending on how 

6 material the event is, you may have no time to react, in 

7 which case the phone call is essentially it. 

8 If we put in pre-defined limits, that suggests 

9 that we can in advance figure out all the combinations 

10 and permutations and all the ways things might misbehave, 

11 and I think our fear of getting it wrong is going to lead 

12 us to artificially high limits, in which case we will 

13 have done a lot of work to not much effect. 

14 What we are all talking about is sort of an 

15 escalation of you hit the 70 percent, you hit the 80 

16 percent, you hit the 90 percent, based on how bad this 

17 is, we are going to give you less and less time to react. 

18 We believe the fear of misfiring, the fear of 

19 de-stabilizing the market because of a combination of 

20 things we hadn't thought of could actually lead us to 

21 making and implementing a set of technology that doesn't 

22 get used the way it should, quite frankly. 

23 MR. GAMBALE: Just a comment. As technologists 

24 up here on the panel, it's not difficult to implement a 

25 process that says I'm going to stop processing if I don't 

0177

1 like something. We do that all the time with some of the 

2 output we send to the industry if we don't think it's in 

3 balance, we think something is wrong with it. 

4 What the panel is saying is this is a 

5 commercial decision. It's a regulatory decision, to 

6 decide to stop processing. DTCC is very far down the 

7 food chain from where the kill switch would be 

8 implemented, but we would support it also. 

9 CHAIRMAN SCHAPIRO: I would disagree with that 

10 a little bit in the sense that I don't think it's just a 

11 commercial decision. I think it's a broader market 

12 integrity and confidence in the markets decision about 

13 whether or not at some point if things are out of 
14 control, trading gets stopped. 

15 MR. JAHANI: I just wanted to add once again 

16 Direct Edge is very supportive of the kill switch model. 

17 We actually even signed a letter that was provided. 

18 Although I want to add technology today is not 

19 the issue. Technologically, this is very easy. I think 

20 technology also can actually provide additional value to 

21 this by looking at the stop signs versus the red lights. 

22 There is a lot of functionalities that can be 

23 embedded in a kill switch. First of all, identifying the 

24 right scenarios, creating the right protocols and all 

25 that stuff can be done, but more so, traveling models 

0178

1 that could be used, et cetera. 

2 It just doesn't need to be every time that 

3 there is a little problem you have to completely stop it. 

4 The phone call, but in reality, that could all be 

5 automated. 

6 Generally speaking, we are very much behind it. 

7 MR. BAYER: It takes active engagement if we 

8 were to agree to do this across the board. Everyone 

9 would have to be actively engaged, and we would have to 

10 continually revise the use cases for the metrics and the 

11 monitoring capabilities that are associated with it. 

12 I think it would take a commitment from the 

13 industry to maintain those use cases and metrics 

14 capabilities. 

15 CHAIRMAN SCHAPIRO: Someone this morning 

16 mentioned the idea of a dynamic kill switch. Is that a 

17 feasible idea? I'm not sure what it meant exactly, other 

18 than I assumed it evolved maybe as market conditions 

19 changed.

20 MR. C. COOK: If I may, to me, the whole idea 

21 of a kill switch sort of implies that it's dynamic. If 

22 you have it at different locations and it's looking at 

23 different thresholds and various feedback loops and 

24 things like that, it is a smarter kill switch, I think, 

25 than just the pure unplug the wire from it. 

0179

1 One of the aspects of the control piece, the 

2 kill switch sort of symbolizes to me the fact that there 

3 is so much going on, and I think to your point, I think 

4 the DTCC is almost a central point that could help feed 

5 back into all this stuff because you have an 

6 authoritative view of what has been traded. 

7 To me, it's that aspect of everything kind of 

8 evolving together. I kind of see it as this ebb and flow 

9 of risk limits are approaching but maybe it is offset 

10 somewhere else, and having that knowledge makes the kill 

11 switch dynamic. To me, it's sort of implied in its 

12 making. 

13 MR. STEINBERG: I think that actually gets back 

14 into what Thomas was just asking a moment ago, which is 

15 what is the commitment to continuously evolve the 
16 thresholds, which is really the dynamic component of it. 

17 They just have to be either set at such an 

18 extreme level that we won't do silly things or they have 

19 to be graduated in such a way as to make sure they are 

20 used in the ways in which they were intended. 

21 MR. JAHANI: I just want to add one more thing. 

22 Please bear in mind nobody today would like to really 

23 tolerate misbehavior, as an Exchange. 

24 If you see that the member firm, just as an 

25 example, is choking up our sessions, you immediately pick 

0180

1 up the phone and talk to them, and we ask them the 

2 questions. 

3 It is not about only one situation. In single 

4 sort of situations where a member firm is causing a 

5 problem, I think we can easily handle that. Those single 

6 threaded situations are easy. 

7 The problem is to look at scenarios where 

8 multiple things can go wrong, multiple things can go 

9 wrong. In other words, if the market behavior is 

10 changing and all of a sudden you see sort of an 

11 indication that something is wrong, it always doesn't 

12 need to be wrong on an overall basis. 

13 These are the scenarios that need to be 

14 clearly defined. If this happens and that happens, then 

15 that's when a kill switch goes off. We need to look at 

16 those situations. 

17 I think to your question of whether or not the 

18 industry is committed, at least I can say basically on 

19 behalf of Direct Edge, I think we are very committed to 

20 that. 

21 MR. STEINBERG: I actually think there has been 

22 violent agreement around the concept of having well 

23 established processes and procedures as opposed to the 

24 informal processes and procedures that we already have 

25 today, and implementing them as part of a larger system 

0181

1 that does include a kill switch. 

2 I think just about everybody here has said yes, 

3 it's a good idea. It's then a matter of how you 

4 implement it. 

5 MS. EWING: Again, going to the layered or 

6 multi-layered approach, when you look at the market 

7 access rules and having that in place and the 

8 requirements to adhere to that, if you look at that 

9 complementing what may happen at an Exchange level, we 

10 are looking at that holistically when we think about 

11 different layers. 

12 I think complying with the market access rules 

13 and having the technology in place to do that is a key 

14 building block. 

15 MR. BLOOM: I want to talk about automated kill 

16 switches also. It's easy for us to sort of imagine -- I 

17 think, Anna, you mentioned earlier potential artificial 
18 intelligence, and fairly sophisticated ways to do it. 

19 If I flash back to Dr. Leveson's comments this 

20 morning, I don't know that any of us would have wanted to 

21 be on the first jet if software was controlling the 

22 flaps, right. 

23 You can imagine a scenario where we start with 

24 some very, very basic controls like we talked about, not 

25 ridiculous, but some very high limits, if you blow 

0182

1 through it too quickly, fine, that's the automation, and 

2 everything else starts manual, and then like what happens 

3 with all the processes and how automated trading even 

4 came into being, right, you refine it from there. 

5 I don't know there is a violent objection to 

6 automation. It's really the speed at which you do it. 

7 MR. STEINBERG: And the combination of 

8 thresholds that you would have to put in place before you 

9 would start to trust the automation. 

10 I doubt that any single factor, net notional, 

11 size of fill's, number of fill's, volume, et cetera, 

12 would make sense. 

13 As I said, at least initially, I agree. Trying 

14 to define all of the different combinations of things 

15 that could go wrong in various ways would be complex. 

16 I wouldn't want to be on that jet. I agree. 

17 MR. KIRILENKO: From sort of the initial 

18 presentation that Professor Markus made and from what you 

19 said, it seems that the sort of fundamental issue is to

20 diminish human oversight of systems, automated systems. 

21 That fundamental issue is not going to go away. There 

22 will be errors and malfunctions and those sorts of things 

23 happening over time. 

24 My question is how do we assign liability for 

25 errors and malfunctions? Who is ultimately liable? Is 

0183

1 it the programmer who put in the bug? Is it the firm? 

2 Is it the Exchange? Where does the liability rest? 

3 Should there be clear delineation of where that is? 

4 How do we provision for that, aside from 

5 testing, monitoring, kill switches, all of that. 

6 Something will still happen. Who is liable and what is 

7 the result of that, what is the outcome. 

8 Lou, you mentioned several times that the words 

9 "risk" and "value impacts," presumably it's in the back 

10 of people's minds when that happens. There are events 

11 that brought down companies very quickly, which became 

12 liable for a few lines of code. 

13 What are your thoughts on this? 

14 MR. STEINBERG: My first thought is it's not a 

15 technology problem. That is a business and legal 

16 question. I'm not sure I'm qualified to answer it. 

17 Again, there is probably more than one thing 

18 that has to go wrong in order to have an incident. There 

19 are latent defects. There are triggering events. There 
20 are a combination of things. The easy stuff, we have 

21 already solved for. In fact, we probably designed around 

22 it. 

23 The concept of assigning liability is a little 

24 worrisome to me as a technologist in part because of the 

25 cultural stuff we talked about before and the importance 

0184

1 that people not operate in a zone of fear, that 

2 technologists not operate in a zone of fear when they are 

3 trying to fix something. You don't want them hiding. 

4 In terms of how you account for that, I'm not 

5 sure I'm the right guy. 

6 MR. R. COOK: I think it is a very good 

7 question, but in defense of our panelists, they were sort 

8 of all vetted for their technology background and

9 experiences and being able to describe technology to non10 technologists, not necessarily to address legal issues. 

11 CHAIRMAN SCHAPIRO: If I could go back to kill 

12 switches, what I'm concerned about is I get the idea we 

13 want a dynamic that maybe if something is going on here 

14 that changes the limits here, and we have lots of 

15 scenarios and lots of possibilities, it feels like we're 

16 adding a lot of complexity. 

17 I wonder again if we are going to end up going 

18 down the path that we talked about this morning where too 

19 much complexity might make it more sophisticated, but 

20 it's going to make it harder to build, harder to manage, 

21 harder for everybody to know what are the rules of the 

22 road, what are the points at which their trading could be 

23 cut off. You surely wouldn't want that to be a surprise, 

24 I don't think, if you're running a firm. 

25 I wonder if you have any thoughts about that, 

0185

1 that we could probably make it really sophisticated and 

2 really specialized, but then have we really advanced the 

3 ball, or is a little bit blunter but simpler a better way 

4 to go. 

5 MS. EWING: I will start. Again, I'm 

6 advocating the view of an Exchange, and I will say an 

7 Exchange that supports this. 

8 That is a key concern. With that complexity, 

9 it leads to grayness in decision making. The ability to 

10 second guess a decision that is made. 

11 That is that framework for decision making, 

12 that framework for what the rules are and what the 

13 thresholds are, really, really key. 

14 We would like it simpler for that very reason. 

15 In the light of day, you need to stand behind the 

16 decision to hit that switch, whether it was automated or 

17 manual. 

18 We couldn't agree with you more that we want to 

19 make it as least complex as possible and very clear. 

20 COMMISSIONER WALTER: Anna, would you also be 

21 in favor, if there were sort of simple parameters set, 
22 which of course wouldn't be right for all scenarios, if 

23 you allowed the discretion to be exercised to un-kill and 

24 to revive in a sense, to un-do the switch for that very 

25 rare case where you really think the world is going to 

0186

1 come to an end if this switch goes off in an automated 

2 fashion? 

3 MS. EWING: The concept about the outreach or 

4 the early warning signals is to ensure that this is a 

5 legitimate issue, especially in high moving volatile 

6 markets where you may want to change your credit limit, 

7 et cetera, because your threshold is too low, so that 

8 becomes a key element of the design of the system. 

9 I think how you recover back from hitting the 

10 switch is something that I think would be very difficult 

11 to do, and what are the unintended outcomes and the 

12 downstream impacts, especially to other participants on 

13 the other side of that series of trades. 

14 Those are some of the things that again we 

15 would like to simplify as much as we can, understanding 

16 it's an important decision. That is why we believe it is 

17 a last resort. That is why we believe it has to be a 

18 multi-layered approach. 

19 I would like to also spend time thinking about 

20 how we can augment what is happening at the broker-dealer 

21 level, market access rules, 2.0, however we want to think 

22 about it. 

23 That has to complement what we are looking to 

24 do at the Exchange level with the kill switch. 

25 MR. STEINBERG: To your point, complexity is 

0187

1 the enemy of availability. The only way to meaningfully 

2 automate in my view would be to in advance define all of 

3 the scenarios, which I think would drive a level of 

4 complexity. 

5 I think baby steps make sense. I think we 

6 could have some very simple blunter thresholds. We have 

7 been talking about some of them here. I think we could 

8 leverage those in a way that sort of ratchets up how fast 

9 we need to react without trying to pre-define every 

10 permutation of every failure mode and then script that to 

11 a point where we have a high degree of complexity. 

12 MR. C. COOK: From my perspective, it's kind of 

13 how we looked at defining functional requirements for any 

14 sort of engineering technology. 

15 With the kill switch, there are so many 

16 different problems that could occur. We kind of have to 

17 figure out what are we trying to accomplish with that 

18 kill switch, and is it technology going awry and becoming 

19 out of control, and that's the only way to shut off 

20 trading, or is it risk limits and other thresholds or a 

21 combination. 

22 I think through some sort of investigation 

23 where we really identify the core problems we're trying 
24 to solve first, with that kill switch, and then some of 

25 those other elements will back off, okay, maybe there are 

0188

1 some new regulations or principles that we can define 

2 that will surround the controls of our technology at the 

3 broker level, at the clearing level, at the other layers, 

4 so we can say these are some standards, how do you 

5 control your technology when it goes crazy or you lose 

6 sight of what's going on. How do you manage those 

7 things. 

8 When I've been through numerous audits and 

9 other things just in terms of the normal course of 

10 business, and more recently on the Series 99 exam, which 

11 some of us may have been involved with, the gap that I 

12 see is this step from going from more human operational 

13 aspects to using technology as the core. 

14 I always look at some of these audit things as 

15 okay, we can define the controls of who has access to the 

16 code and who rolls things out, but not once did it ask 

17 what happens when the system goes crazy and you lose 

18 access to it. 

19 These are some of the things that are kind of 

20 low hanging fruit, I think, that if we put some of that 

21 out there as clearly defined principles or best practices 

22 or what not, how do we manage our technology now that 

23 it's such a critical part of everything that we do. 

24 That is a little bit of how I look at it. 

25 COMMISSIONER WALTER: Doesn't that to a certain 

0189

1 extent go against what in some sense is the core lesson 

2 or one of the core lessons of the financial crisis, which 

3 is the thing that is going to go wrong is the thing you 

4 don't anticipate. 

5 If you have to figure out what the scenarios 

6 are in order to define it, in some sense, I think it will 

7 fail. I think one of the advantages of a blunter tool is 

8 that it may not be the right thing in a few instances, 

9 but at least it confines the damage and doesn't require 

10 you to identify first what's going to go wrong. 

11 MR. C. COOK: I agree. I think the risk 

12 management approach would look at it in terms of what are 

13 the ramifications we are trying to solve first, not 

14 necessarily is there trading going crazy, but what are we 

15 trying to prevent. 

16 With those kill switches, are we really trying 

17 to prevent massive market upheaval or are we trying to 

18 prevent the broker-dealer from losing all their money 

19 because something went wrong. 

20 I think looking at it from that perspective 

21 gives a better basis as opposed to defining every single 

22 idea. There are a million test cases that we could 

23 really look for, and that sort of goes back to one of the 

24 statements we were talking about earlier in terms of how 

25 we approach risk in the systems from a technology 
0190

1 perspective, and what should it do, what shouldn't it do. 

2 We can't enumerate all of those cases where 

3 things will go wrong. There are too many 

4 interdependencies between operating systems and intel 

5 CPUs, libraries, and things of that nature. 

6 It is how do we identify when something goes 

7 wrong, and then what are we going to do about it. Kind 

8 of in that realm. 

9 MR. STEINBERG: Just to add to that, the QA is 

10 really designed to test cases, really designed to 

11 identify and mitigate your latent defects, and 

12 operational processes are designed to be controls around 

13 triggering events. 

14 To your point, you still will have unforeseen 

15 incidents that neither of those captured, where the only 

16 thing you can do is mitigate the impact, in this notion 

17 of latent defects trigger an impact. 

18 What do we do when something bad still happens, 

19 and kill switches are one form of impact mitigation. 

20 There are other forms as well, since a lot of incidents 

21 are triggered by change. 

22 I mentioned before we are implementing a model 

23 that allows us to stage changes and then if for some 

24 reason they are not behaving the way we expected, to fail 

25 back to a set of systems that have not been touched, and 

0191

1 in fact, give us a fast fail back capability without 

2 trying to figure out what went wrong, just recover, 

3 mitigate and control the impact that way. 

4 Kill switches are certainly one way to mitigate 

5 impact when QA didn't find the latent defects, when intel 

6 didn't prevent the trigger, but there are probably other 

7 ways that are worth investigating as well. 

8 MR. R. COOK: We are getting near the end of 

9 our time. I just wanted to ask one quick wrap up 

10 question. Are there any other issues you think we should 

11 be thinking about in terms of reducing risk to the 

12 system. 

13 I know some of the comment letters talked about 

14 the settlement cycle, for example, or locked in trades. 

15 I will open it up to anyone. It is something 

16 we haven't talked about yet. We don't have time to 

17 develop it, but at least to have it on our radar screen. 

18 MR. GAMBALE: One of the points earlier was we 

19 do think DTCC receives the most trades in real time, 

20 although not all, and we think it is important for risk 

21 mitigation and flow of the industry for us to be able to 

22 see all the trades as quickly in real time as possible. 

23 This would help us, as Chad pointed out 

24 earlier, if we could be a participant in the kill switch 

25 feedback to the Exchanges and trade sources. 

0192

1 MS. EWING: I would just recap really quickly 
2 some of what was discussed this morning and earlier, and 

3 that is the real time nature of our markets, and the more 

4 we can do real time surveillance, monitoring, kill switch 

5 decisions, that is the world we live in. 

6 As we think about the people, the processes, 

7 the technology, we have to evolve on all three fronts. 

8 I think the collaboration that we have as an 

9 industry, this is an example, a latest example of us 

10 coming together, the working group, and advancing the 

11 idea of the kill switches, et cetera. 

12 I have full confidence and we are fully 

13 supporting industry collaboration, and that collaboration 

14 is absolutely key to anything we advance, including with 

15 the SEC, our regulator, and having those dialogues, just 

16 like we do in information security. 

17 There is a real tight community between the 

18 regulators, the agencies, other financial institutions, 

19 and we come together in formal mechanisms. We share 

20 information. There is no such thing as competition 

21 between us because we all have a common goal, and that is 

22 resiliency and integrity of our markets. 

23 That is a model that is in place today that we 

24 can continue to build upon, and as we look at technology 

25 and we look at some of the mechanisms we can improve 

0193

1 upon, whether it's the monitoring or architecture, et 

2 cetera, we are looking at the commingling of what we have 

3 advanced. There are a lot of great tools in the 

4 information security space that we are now deploying into 

5 our market systems and into the rest of our architecture. 

6 That is another area that we are advancing. We 

7 are working with thought leaders. We are working with 

8 the Software Engineering Institute at Carnegie-Mellon, to 

9 really advance some of those concepts around how to 

10 monitor and create resilient systems through software, 

11 through controls, through more technology. I would just 

12 add that as a consideration. 

13 MR. R. COOK: Thank you. Again, thank you all 

14 for participating today. We really appreciate you 

15 spending the afternoon with us and sharing your thoughts 

16 and experiences. 

17 You have given us a lot to think about. I will 

18 reiterate that our comment file remains open. Anyone is 

19 free to submit their recommendations to us either 

20 building on anything that was presented at one of these 

21 panels or other ideas folks may have. 

22 Thank you very much for your participation, and 

23 that is the end of our roundtable. 

24 (Whereupon, at 4:01 p.m., the roundtable was 

25 concluded.) 

0194

1 PROOFREADER'S CERTIFICATE 

2 

3 In The Matter of: TECHNOLOGY AND TRADING: PROMOTING 
4 STABILITY IN TODAY'S MARKETS 

5 File Number: OS-4-652 

6 Date: Tuesday, October 2, 2012 

7 Location: Washington, D.C. 

8 

9 This is to certify that I, Donna S. Raya, 

10 (the undersigned), do hereby swear and affirm that the 

11 attached proceedings before the U.S. Securities and 

12 Exchange Commission were held according to the record and 

13 that this is the original, complete, true and accurate 

14 transcript that has been compared to the reporting or 

15 recording accomplished at the hearing. 

16 

17 _______________________ _______________________ 

18 (Proofreader's Name) (Date) 

19 

20 

21 

22 

23 

24 

25 

0195

1 REPORTER'S CERTIFICATE 

2 I, Jon Hundley, reporter, hereby certify that the 

3 foregoing transcript of 194 pages is a complete, true and 

4 accurate transcript of the testimony indicated, held on 

5 October 2, 2012, at Washington, D.C. in the matter of: 

6 TECHNOLOGY AND TRADING: PROMOTING STABILITY IN TODAY'S 

7 MARKETS. 

8 

9 I further certify that this proceeding was recorded by 

10 me, and that the foregoing transcript has been prepared 

11 under my direction. 

12 

13 

14 Date:__________________________ 

15 Official Reporter:__________________________ 

16 Diversified Reporting Services, Inc. 

17 

18 

19 

20 

21 

22 

23 

24 

25