How to Easily Fetch Binance Historical Trades Using Python
By: Thiago Candido
Tags: Blockchain, Programming, Python, Cryptocurrency, Crypto

Coding Time

Parsing the arguments

The script will use the following arguments:

symbol : The symbol of the trading pair, defined by Binance. It can be queried here, or it may be copied from the URL of the Binance web app, excluding the _ character.

Remove the ‘_’ from the last part of the URL and you get the symbol

starting_date and ending_date : Self-explanatory. The expected format is mm/dd/yyyy , or, in Python slang, %m/%d/%Y .

To get the arguments, we’ll use the built-in sys (nothing too fancy around here), and to parse the date, we will be using the datetime library.

symbol = sys.argv[1]

starting_date = datetime.strptime(sys.argv[2], '%m/%d/%Y')

ending_date = datetime.strptime(sys.argv[3], '%m/%d/%Y') + timedelta(days=1) - timedelta(microseconds=1)

We are adding one day and subtracting one microsecond so that the ending_date time portion is always at 23:59:59.999 , making it more practical to get same-day intervals.

Fetching trades

With Binance’s API and using the aggTrades endpoint, we can get at most 1,000 trades in one request, and if we use start and end parameters, they can be at most one hour apart. After some failures, by fetching using time intervals (at some point or another, the liquidity would go crazy and I would lose some precious trades), I decided to try the from_id strategy.

The aggTrades endpoint is chosen because it returns the compressed trades. In that way, we won’t lose any precious information.

Get compressed, aggregate trades. Trades that fill at the same time, from the same order, with the same price will have the quantity aggregated.

The from_id strategy goes like this: We are going to get the first trade of the starting_date by sending date intervals to the endpoint. After that, we will fetch 1,000 trades, starting with the first fetched trade ID. Then, we will check if the last trade happened after our ending_date . If so, we have gone through all the time period and we can save the results to file. Otherwise, we will update our from_id variable to get the last trade ID and start the loop all over again.

Ugh, enough talking, let’s code.

Fetching the first trade ID

First, we create a new_end_date . That’s because we are using the aggTrades by passing a startTime and an endTime parameter. For now, we only need to know the first trade ID of the period, so we are adding 60 seconds to the period. In low liquidity pairs, this parameter can be changed because there is no guarantee that a trade occurred in the first minute of the day that’s been requested.

Then, parse the date using our helper function to convert it to a Unix millisecond representation by using the calendar.timegm function. The timegm function is preferred because it keeps the date in UTC.

def get_unix_ms_from_date(date):

return int(calendar.timegm(date.timetuple()) * 1000 + date.microsecond/1000)

The request’s response is a list of trade objects sorted by date, with the following format:

So, as we need the first trade ID, we will be returning the response[0]["a"] value.

Main loop

Now that we have the first trade ID, we can fetch trades 1,000 at a time, until we reach our ending_date . The following code will be called inside our main loop. It will perform our request using the from_id parameter, ditching the startDate and endDate parameters.

And now, here’s our main loop, which will perform the requests and create our DataFrame .

We check if the current_time that contains the date of the latest trade fetched is greater than our to_date , and if so, we:

fetch the trades using the from_id parameter

parameter update the from_id and current_time parameters, both with information from the latest trade fetched

and parameters, both with information from the latest trade fetched print a nice debug message

debug message pd.concat the trades fetched with the previous trades in our DataFrame

the trades fetched with the previous trades in our and sleep a little so that Binance won’t give us an ugly 429 HTTP response

Cleaning and saving

After assembling our DataFrame , we need to perform a simple data cleaning. We will remove the duplicates and trim the trades that happened after our to_date (we have that problem because we’re fetching in chunks of 1,000 trades, so it’s expected that we get some trades executed after our target end date).

We can encapsulate our trim functionality:

def trim(df, to_date):

return df[df['T'] <= get_unix_ms_from_date(to_date)]

And perform our data cleaning:

df.drop_duplicates(subset='a', inplace=True)

df = trim(df, to_date)

Now, we can save it to file using the to_csv method:

filename = f'binance__{symbol}__trades__from__{sys.argv[2].replace("/", "_")}__to__{sys.argv[3].replace("/", "_")}.csv'

df.to_csv(filename)

We can also use other data storage mechanisms, such as Arctic.