Polygon.io Review — Retrieving Historical Tick Data | by Mario Emmanuel | Towards Data Science

During last months one of my main objectives and one of the most time-consuming tasks has been to extract edges from tick data for Intraday operations.
Within this context, a quality data feed becomes extremely important. I have started to use Polygon.io to get tick data from SPY as I want to test a set of discretionary strategies I have been learning during the last months. The strategies are intended to be operated on Futures but I want to evaluate if they work with other assets/instruments such as the SPY ETF and Forex instruments.
This article covers the retrieval of historical data via their REST API. I will also briefly cover the size of the data involved.
The first thing you need to understand is that tick data usually involves two separate data feeds. One covers the top of book, including best bid and best ask values and the other one covers the actual trades. In equities (SPY is an ETF, but ETFs are traded as equities), there are different exchanges where trade can take place. That means that there is not a central point where the trades take place. The US has currently 13 listed exchanges. Not all assets are traded in every exchange, but they are neither traded in just one. There are also dark pools and private exchanges, where Institutional players can cross orders and large brokers can also cross orders among customers.
That means that the simplified view of a single, unified, continuous and unique stream of quotes and trades does not exist in the real world.
All market information is consolidated by SIPs (Securities Information Processors), which are regulated entities that consolidate and provide a unified data feed on equities. They define a clear and deterministic set of rules on how trades are consolidated. The two operating SIPs in the US are The Consolidated Tape Association (CTA) and the Unlisted Trading Privileges (UTP). Those SIPs consolidate NYSE and NASDAQ listed symbols respectively. Pretty good information on how they work can be found in this Blog entry.
While this information is not important for a retail trader (very few people understands how all this mumbo jumbo works), there is a couple of important takeaways to know:
Stocks are traded in different places, so they are not fully centralised. What you see with a quality real-time data feed is just a decent proxy on what it is actually happening in the market.Different timestamps relates information at the SIP and at the exchange. They are not relevant for the retail but are relevant to relate Bid/Ask prices with Trades (such as required by OrderFlow and Delta analysis).
Stocks are traded in different places, so they are not fully centralised. What you see with a quality real-time data feed is just a decent proxy on what it is actually happening in the market.
Different timestamps relates information at the SIP and at the exchange. They are not relevant for the retail but are relevant to relate Bid/Ask prices with Trades (such as required by OrderFlow and Delta analysis).
The situation is the same for the Depth of Book, but I will not cover that because at this moment I am not analysing Depth of book in any of my strategies.
In order to get historic quotes, use /v2/ticks/stocks/nbbo/{ticker}/{date} request
Historical data can be retrieved from Polygon via their REST API interface. The server will submit a maximum of 50,000 records for each request. While 50,000 records for OHLC daily data represents more than 200 years of stock market history, it is just a small fraction of the daily data volume for tick data.
Basic date and ticker parameters are included as part of the URL:
ticker:string, ticker name (i.e. SPY for SPDR S&P500 ETF)date:string, date in YYYY-MM-DD format. Notice that the API assumes Eastern Time by default, so 2020-06-15 means 2020-06-15 00:00 New York time.
After a long (and often painful) work with timezones, I have concluded that an easy and consistent way is to store and process all your market data in ET. It is widely used as reference timezone for market data (sometimes even for European assets/instruments) and it is also the iconic time reference for financial markets (NYSE RTH opening is an observed daily event for everyone). This does not apply to crypto data, because it operates 24x7 and you will face issues when switching from daylight saving to non-daylight saving. In that case, it is recommended to use GMT.
The API is fully documented here, but I highlight/comment on the following parameters. Notice that these are specified as HTTP GET parameters and not as part of the URL.
timestamp:long value, represents the first epoch timestamp (with nanoseconds resolution) for which data is requested. timestampLimit:long value, represents the last epoch timestamp (with nanoseconds resolution) for which data is requested, provided that the hard limit of 50,000 (or the custom limit) is not reached first. Note that this timestamp is returned, so the value is included.limit:integer value, maximum number of records to be retrieved, 50,000 if not specified.apiKey:string, your API key, required in every request.
As we have the limit of the 50,000 items, it is needed to perform pagination. To perform pagination we will simply use the timestamp parameter. The algorithm is pretty simple: if we are receiving 50,000 items we assume that an additional request is required, we specify the latest received timestamp as a parameter in the next request. We do that until the last request receives less than 50,000 items. That means that there are no more requests to be done.
Three specific details must be taken into consideration:
It is a good idea to set up timestampLimit as the next day’s 00:00 time. As API includes timesetampLimit as a valid returned item, we need to exclude later that value from the returned results.
The timestamp used for the successive request is included twice, as it is the boundary between the n and the n+1 requests. Hence it is important to remove it from the n request when an n+1 request is planned.
It might be the case that we receive 50,000 items and those are exactly all the data available. In that case, our algorithm will request a new batch which shall return only one result. This is not an issue and it is part of how the algorithm does pagination.
An alternative would be to add a 1 nanoseconds to the timestamp request for successive requests; this would theoretically avoid the need for duplicates in subsequent requests. As I am not 100% sure that two records can be received for the same nanosecond epoch timestamp, I have decided not to follow that path, as the previous one can handle that scenario of multiple records for the same timestamp.
The very same algorithm is used both for Quotes and Trades (and could be coded in a separate shared method/routine).
In order to get historic trades, use /v2/ticks/stocks/trades/{ticker}/{date} request
As already mentioned, historical trades can be retrieved similarly. The same conditions and parameters apply.
Historical trades shall be almost always stored and processed locally to avoid overhead. Higher timeframes can be requested on-demand, but requesting months of historical tick data on-demand on every backtesting iteration is simply not practical.
I always try to store an easy to read and easy to parse file with raw data, using ASCII text files if possible. In this example, we create two files (quotes and trades).
user@almaz:~/NetBeansProjects/SealionPolygonAdapter % java -jar dist/SealionPolygonAdapter.jar159238734271500798615923935052033437761592398291430866793...1592419244447798705159242198141006489615924234856001943781592423485600194378user@almaz:~/NetBeansProjects/SealionPolygonAdapter % ls -lh data/*-rw-r--r--  1 user  user   399M Jun 21 19:04 data/SPY.2020-06-17.quote.json-rw-r--r--  1 user  user    54M Jun 21 19:05 data/SPY.2020-06-17.trade.jsonuser@almaz:~/NetBeansProjects/SealionPolygonAdapter % head data/SPY.2020-06-17.quote.json {"p":313.9,"P":315,"q":9801,"c":[1,81],"s":2,"S":1,"t":1592380800048084434,"x":12,"X":11,"y":1592380800047963904,"z":1}{"p":313.9,"P":315,"q":9901,"c":[1,81],"s":2,"S":5,"t":1592380800048834673,"x":12,"X":11,"y":1592380800048768512,"z":1}{"p":313.9,"P":315,"q":10001,"c":[1,81],"s":2,"S":6,"t":1592380800048839409,"x":12,"X":11,"y":1592380800048774400,"z":1}...
The data is stored as one JSON object per line which can be easily parsed (notice that it is also inefficient in terms of storage and performance). It uses around 450Mb per day, which is around 100Gb of data per year. If you are storing many assets, one key factor would be to pay attention to integer compression algorithms. Using proprietary and optimised databases will save space, although as a trade-off it has a cost in terms of development and debugging effort and an increase in operational complexity. I still like to have a plain backup of readable data, even if I further transform the data to a more performance-friendly format.
We can also notice the amount of data that every trading day involves:
user@almaz:~/NetBeansProjects/SealionPolygonAdapter % wc data/SPY.2020-06-17.quote.json  3060489 3060489 418579482 data/SPY.2020-06-17.quote.jsonuser@almaz:~/NetBeansProjects/SealionPolygonAdapter % wc data/SPY.2020-06-17.trade.json  435989  435989 56996300 data/SPY.2020-06-17.trade.json
SPY has an outstanding figure of 3 millions of quote ticks and more than 400K trade ticks in a single day.
In order to use the data in any backtesting or custom tool, further processing is likely required, but as mentioned at the beginning of the article, storing data in simple ASCII readable data is useful as a permanent backup. This will simplify development and conversion time and will decouple data historical provider from our analysis toolchain.
A simple but efficient way of improving this would be to compress both archives. In such case we get:
user@almaz:~/NetBeansProjects/SealionPolygonAdapter % gzip data/SPY.2020-06-17.quote.jsonuser@almaz:~/NetBeansProjects/SealionPolygonAdapter % gzip data/SPY.2020-06-17.trade.jsonuser@almaz:~/NetBeansProjects/SealionPolygonAdapter % ls -lh data/*-rw-r--r--  1 user  user    54M Jun 21 19:04 data/SPY.2020-06-17.quote.json.gz-rw-r--r--  1 user  user   9.2M Jun 21 19:05 data/SPY.2020-06-17.trade.json.gz
By just using the standard gzip tool we get just 64Mb/day, which requires around 16Gb per year. A much more friendly storage size that can be easily fulfilled without special resources.
Compression and decompression take time, so this shall be used only as a permanent data backup.
A basic version of the source code to retrieve historical quotes (both quotes & trades) is included as reference below. Notice that the code is beta and not optimised, it is intended to show a use case in Java that clarifies how to access historical tick data. It can be easily ported to other languages.
The data is retrieved in JSON format, which is a format that makes debugging easy. The tradeoff is that that JSON is far from being an efficient format. For a historical data retrieval API, this is in my opinion not relevant, especially if data is retrieved in batch mode for backtesting purposes. In that scenario, the simplicity and maintainability provided by JSON are clear advantages.
For this particular implementation, I have used simple-json v1.1.1 as a companion library to parse JSON. This library is no longer maintained in its version 1, but it is a fully working library, extremely simple to use and lightweight.