Package: iex-parser
iex_parserParser for IEX pcap DEEP (1.0) and TOPS (1.5, 1.6) files.OverviewAt the time of writing the IEX exchange provides two file downloads for
historical data: DEEP and TOPS. This data is provided as apcapfile
which is a dump of the network activity.This package provides an API for extracting the data from these files.InstallationInstall from PyPi.pipinstalliex_parserExampleThe following code processes the TOPS sample file downloaded from IEX. Note the
file doesn't have to be unzipped. For version 1.6 useTOPS_1_6, for 1.5 useTOPS_1_5.fromiex_parserimportParser,TOPS_1_6TOPS_SAMPLE_DATA_FILE='data_feeds_20180127_20180127_IEXTP1_TOPS1.6.pcap.gz'withParser(TOPS_SAMPLE_DATA_FILE,TOPS_1_6)asreader:formessageinreader:print(message)The result looks like this:{'type':'trading_status','status':b'T','timestamp':datetime.datetime(2018,1,27,15,23,40,490473,tzinfo=datetime.timezone.utc),'symbol':b'SPEM','reason':b''}{'type':'trading_status','status':b'H','timestamp':datetime.datetime(2018,1,27,15,23,42,95642,tzinfo=datetime.timezone.utc),'symbol':b'INCO','reason':b'NA'}{'type':'trading_status','status':b'H','timestamp':datetime.datetime(2018,1,27,15,23,42,852349,tzinfo=datetime.timezone.utc),'symbol':b'CHSCN','reason':b'NA'}{'type':'price_level_update','side':b'S','flags':1,'timestamp':datetime.datetime(2018,1,27,15,23,44,856983,tzinfo=datetime.timezone.utc),'symbol':b'ATLO','size':8755,'price':Decimal('38.95')}{'type':'price_level_update','side':b'S','flags':0,'timestamp':datetime.datetime(2018,1,27,15,23,44,856983,tzinfo=datetime.timezone.utc),'symbol':b'ATLO','size':37222,'price':Decimal('48')}{'type':'price_level_update','side':b'S','flags':1,'timestamp':datetime.datetime(2018,1,27,15,23,44,856987,tzinfo=datetime.timezone.utc),'symbol':b'ATLO','size':8958,'price':Decimal('38.95')}{'type':'price_level_update','side':b'S','flags':0,'timestamp':datetime.datetime(2018,1,27,15,23,44,856987,tzinfo=datetime.timezone.utc),'symbol':b'ATLO','size':37019,'price':Decimal('48')}The following code processes the DEEP sample file downloaded from IEX.fromiex_parserimportParser,DEEP_1_0DEEP_SAMPLE_DATA_FILE='data_feeds_20180127_20180127_IEXTP1_DEEP1.0.pcap.gz'withParser(DEEP_SAMPLE_DATA_FILE,DEEP_1_0)asreader:formessageinreader:print(message)SpeedBecause the data is distributed as a dump of network packets, there are a lot of "empty"
packets. These take time to read and slow the delivery of the real data. To handle this
the packets are read on a separate python thread and queued. The size of the queue is an
optional parameter to theParser, and has been set by experimentation to 25000.The main question I get is: can it go any faster?The short answer is no. However the reason for the slowness is the time spent
reading and skipping network data in thepcapfile.The solution is to convert the downloadedpcapfiles intocsvorJSON.Command line toolsThere are command line tools that takes a downloaded file and converts it
to csv files or a JSON file.iex-to-csv$iex-to-csv-i<input-file>-o<output-folder>[-s][-t<ticker>...][-c]The input file must be as downloaded from IEX. This-sflag can be used to
suppress the progress printing. The-tflag can be used to select specific
tickers. The-cflag cause the ordinal to be reset when the timestamp changes,
rather than monotonically increasing. A file for every message type is produced.For example:$iex-to-csv-i~/data/raw/data_feeds_20200305_20200305_IEXTP1_DEEP1.0.pcap.gz-o~/data/csviex-to-json$iex-to-json-i<input-file>-o<output-path>[-s][-t<ticker>...]The input file must be as downloaded from IEX. This-sflag can be used to
suppress the progress printing. The-tflag can be used to select specific
tickers. A single file is produced containing a JSON message per line.For example:$iex-to-json-i~/data/raw/data_feeds_20200305_20200305_IEXTP1_DEEP1.0.pcap.gz-o~/data/json/There is a helper function to load this data:frompathlibimportPathfromiex_parser.iex_to_jsonimportload_jsonINPUT_FILENAME=Path('data_feeds_20200305_20200305_IEXTP1_DEEP1.0.json.gz')forobjinload_json(INPUT_FILENAME):ifobj['type']=='trade_report':print(obj)MessagesThe messages are returned as dictionaries.Security Directive{'type':'security_directive','flags':int,'timestamp':datetime.datetime,'symbol':bytes,'round_lot_size':int,'adjusted_poc_close':decimal.Decimal,'luld_tier':int}Trading Status{'type':'trading_status','status':bytes,'timestamp':datetime.datetime,'symbol':bytes,'reason':bytes}Operational Halt{'type':'operational_halt','halt_status':bytes,'timestamp':datetime.datetime,'symbol':bytes}Short Sale Price Test Status{'type':'short_sale_price_test_status','status':int,'timestamp':datetime.datetime,'symbol':bytes,'detail':bytes}Quote Update{'type':'quote_update','flags':int,'timestamp':datetime.datetime,'symbol':bytes,'bid_size':int,'bid_price':decimal.Decimal,'ask_size':int,'ask_price':decimal.Decimal}Trade Report{'type':'trade_report','flags':int,'timestamp':datetime.datetime,'symbol':bytes,'size':int,'price':decimal.Decimal,'trade_id':int}Official Price{'type':'official_price','price_type':bytes,'timestamp':datetime.datetime,'symbol':bytes,'price':deccimal.Decimal}Trade Break{'type':'trade_break','flags':int,'timestamp':datetime.datetime,'symbol':bytes,'size':int,'price':decimal.Decimal,'trade_id':int}Auction Information{'type':'auction_information','auction_type':bytes,'timestamp':decimal.Decimal,'symbol':bytes,'paired_shares':int,'reference_price':decimal.Decmal,'indicative_clearing_price':decimal.Decimal,'imbalance_shares':int,'imbalance_side':bytes,'extension_number':int,'scheduled_auction_time':datetime.datetime,'auction_book_clearing_price':decimal.Decimal,'collar_reference_price':decimal.Decimal,'lower_auction_collar_price':decimal.Decimal,'upper_auction_collar_price':decimal.Decimal}Price Level Update{'type':'price_level_update','side':bytes,'flags':int,'timestamp':datetime.datetime,'symbol':bytes,'size':int,'price':decimal.Decimal}Secrity Event{'type':'security_event','security_event':bytes,'timestamp':datetime.datetime,'symbol':bytes}