CFTC

TAC Meeting

June 3, 2014

CFTC Data Landscape and

Update on Joint Efforts on Data Quality

Jorge Herrada, Office of Data and Technology
Commodity Futures Trading Commission

• Data at CFTC

– Daily data volumes

– Nine categories of data

– Details of data categories

– Data loading challenges

– Trends in data volumes

• SDR Data Harmonization Update

Agenda

2
Commodity Futures Trading Commission

(not to scale)

SDR Data

Margin

& Collat

Data Feeds 

(approximate # of million records per day

CFTC is currently loading)

Market Data

0.2M / Day

Margin & 

Collateral

3M / Day

Valuation / Stress Test

0.1M / Day

Fin Statement / Capitalization

0.1M / Day

Org / Product Data

0.1M / Day

FERC Data

0.33M / Day

Time & Sales Futures Data 

(top of the order book)

240M / Day 

(not to scale, at full scale 

this bubble would be 8 feet across)

Enforcement

.2 – 1TB / Week

(not to scale, scale = 10 feet) 2.5M records / inch

 Commodity Futures Trading Commission

Futures & Swaps

Position Data

16M / Day

Futures Intraday Trades / 

Swaps Events

10M / Day (Note: adding order 

messages would be orders of 

magnitude greater, adding 200 –

300M records per day. 

At scale, adds another 

7 - 10 feet to 

the diameter)
Commodity Futures Trading Commission

1. Intraday Trades / Events Data – All trades on futures exchanges and all events 

reported to SDRs

2. Time & Sales Data – Top of the order book at time futures trade executes.

3. Position Data - End-of-day long/short contract positions for futures and options or 

notional amounts for swaps.

4. Market Data - Real-time or end-of day market data (e.g. settlements, open interest, 

volume, bid/ask, last price, etc.)

5. Margin & Collateral Data - Daily or intraday initial margin, maintenance margin, 

variation margin, or collateral on deposit

6. Valuation & Stress Test Data - Daily data used to value futures or swaps positions 

(e.g. IRS curve/discount factor), or stress test results from what-if scenario analysis

7. Financial Statement / Capitalization Data - Daily or monthly data used from 

financial statement submissions

8. Organization & Product Data – Reference data about Organizations and Product 

specifications

9. Enforcement Data – Investigation and case files (including subpoenaed information 

and audio tapes)

Nine Categories of Data

4
Commodity Futures Trading Commission

• Trade Data

• CME Tag 50

• CME TCR Broker Ref

• CME Bulletin

• Time & Sales Data

• T&S Spreads

• LT Pos Data (part 17)

• Clearing Member Positions 

(part 16)

FERC LT Position export

• Market Data (part 16, price, 

volume, OI)

FERC market export

MDS – Reuters, Futures 

Source, Bloomberg

• CME Customer Margin

• SPAN Margin, Risk Arrays • ICE Non-Span Margin 

Engine

• Winjammer / RSR Loaders

• Form – Tips Complaints & 

Referrals

• Subpoenas, massive data 

loads

• Product Reference Files

• Forms – DCO, DCM

• Form OCR

• Form 102 (Spec Acct)

• Form 40

• Form 204, 304

• Part 45 (DDR only) SDR Portals (DDR, ICE, CME, 

Bloomberg)

• DCO data (part 30)

• Part 45 (DDR only)

• Part 20 Phys Commodities

• DCO Pos Data (Part 39)

SDR Portals (DDR, ICE, CME, 

Bloomberg)

MDS – Reuters, Futures 

Source, Bloomberg

Markit

• DCO Acct (margin & 

collateral)

• CDS margin & stats

• CME IRS Transfer receipts, 

margin, XV, positions. 

• IRS CME val., delta ladders, 

settlements, curves, & disc 

factors

• IRS LCH – delta ladders, 

margin, valuation

 ICC Margin API

• Fin Statement data

• Monthly excess net capital

 NFA Daily Segregated 

funds

• Forms – Tips, Complaints & 

Referrals

• Subpoenas, massive data 

loads

• Forms – SEF

• Forms – SDR

 CICI / LEI (part 45)

 NFA Reg info

 Global LEI repository
Commodity Futures Trading Commission

0

20

40

60

80

100

120

140

160

Enterprise Infrastructure

(Voicemail, email, etc.)

eLaw Structured Market Data (Large

Trader, Transaction Data, etc.)

Enterprise Application Data

CFTC Network Storage Used by Program (TB)

6

Enforcement servers represent 39% of all data stored, 

whereas structured data represents 38% of the data stored.
Commodity Futures Trading Commission

CFTC Servers by Program

7

0

20

40

60

80

100

120

140

Enterprise Infrastructure

(Voicemail, email, etc.)

eLaw Structured Market Data (Large

Trader, Transaction Data, etc.)

Enterprise Application Data

Enforcement (eLaw) servers represent 25% of all servers, 

whereas structured data represents 11% of the servers
Commodity Futures Trading Commission

Legal Hold

Management

8

eDiscovery Review

Case

Tracking

FOIA

Tracking

eDiscovery

Processing

Audio

Search

Digital

Forensics

Trial

Presentation

Legacy

eDiscovery 

Review

Financial

Statement

Analysis

Major eLaw 

Applications

(All of these apps in aggregate require 

a significant # of servers)

Highly standardized data, business 

functions and software products.

Bubble sizes relative to importance to eLaw suite
Commodity Futures Trading Commission

eLaw staffing and data trends. Consistent software, data, and business 

functions are in place, so IT can ramp up data ingest with minor staffing implications.

9
Commodity Futures Trading Commission

Status of Structured Data Loading at CFTC

Increasing scope and 

complexity of structured data 

files (non-eLaw)

In 2013 we added loaders to handle 65 

new types of data. Prior to 2013, we 

only handled 51 in total. In the last 12 

months, we have added loaders for 77 

new types of data. A 166% increase...in 

just 12 months

Each new data set requires data 

standardization, data architecture, 

custom loaders, data quality checks, 

monitoring, and feedback to submitters. 

Unlike eLaw, each new data set is a 

major project unto itself, analogous to 

standing up a new factory production 

line. 

Data standards and harmonization are 

critical. 

10

Loaders

0

20

40

60

80

100

120

140

2010 2011 2012 2013 2014

Types of Data Files

0

5

10

15

20 Contractors and FTEs directly involved in structured data ingest

Contractor

FTE
Commodity Futures Trading Commission

Data Loading is now a 24hr per day operation

Challenges in Loading Data at CFTC

11
Commodity Futures Trading Commission

• Phase I Harmonization implemented by SDRs for the credit asset 

class

• Phase II Harmonization (credit) in final stages of ODT review 

• Scheduled bilateral meetings with SDRs for Phase II

o Discuss progress on SDR harmonization action plans

o Share Commission’s independent data quality analyses with SDRs

o Discuss exception reporting from SDRs to data submitters

• Plan a joint SDR meeting this summer for Phase II Harmonization

• Joint effort with Office of Financial Research (ORF)

o OFR to assist with harmonizing IRS asset class

o OFR to also assist Office of the Chief Economist with IRS research 

o IRS research to include swaps and futures data

Harmonization Update

12