The Open-Source Goldmine of Crypto-Markets Datasets
By: Julian Molina
Tags: Data Analysis, Cryptocurrency, Data Visualization, Data Science, Data Processing

I should have known from that first conversation that I was about to get dragged into yet another multi-years long project. It all started back in the summer of 2017.

Luis Fernando Molina, my brother, had been toying for weeks with a visual interface he envisioned would be the foundation for a new kind of trading bots platform. He insisted that being able to visualize entire datasets was at the core of creating trading intelligence.

To me, data visualization and trading bots in the same sentence seemed counter-intuitive. However, his next statement made it obvious…

Bots — he said — don’t need data visualization, but humans building bots do.

Little did I know that we would spend the next three years building a system of titanic proportions! Boy was I dragged into a roller coaster ride!

We launched a startup, built a team, and — at some point — decided to radically change plans and go open-source instead.

I will get to pen that story one day…

But today, I’m writing to share with the data science community the free and open-source data-mining infrastructure we have created for Superalgos, which may be of use to people doing crypto-markets research, building trading AIs, or looking for a visual-scripting solution to process data or run backtests.

Like quants in the community have rightly pointed out, the infrastructure not only serves the purpose of providing our systematic strategies with high-quality Technical Analysis datasets…

Superalgos Data Mines solve the software and data engineering needs of data scientists — like extracting, collecting, cleaning, and storing market data — as well as some of the data analysis needs — like transforming, aggregating, and processing data.

The Data Mines infrastructure was built as the information leg of an algorithmic crypto-trading platform, but, to be precise, it may handle all sorts of data, provided that the right sensors were added.

What follows is the bird’s-eye view of what we’ve built so far, and what you get right out-of-the-box, free of charge, no ads, no collection of personal information, no paid pro-version… just good, old-school open-source software you download and run on your machine.

System Architecture

We wanted a flexible visual environment that would lower the barrier of entry and help the less technically-minded users make the most out of the system. At the same time, we wanted a robust and reliable backend, worthy of a mission-critical financial system.

At the highest level, the system is divided into a frontend web app running an animated HTML5 canvas on the browser, and a backend consisting of a set of Node JS processes.

The codebase is distributed with the system and constitutes a fully functional development environment right out of the package. The system runs uncompiled on the actual code.

Data mines are designed to sustain a real-time data-mining and data-processing operation that feeds elaborate information to trading bots, but may be used to download data and process it offline as well.

Exchange Connectivity

We started out working solely with Poloniex, the top-of-the-food-chain crypto exchange back in mid-2017. By the time the core of the system had gone through a few iterations, Poloniex was nowhere to be found in the top-20.

Boy, do things change fast in crypto-land! It was time to open up the game to more exchanges…

Crypto exchanges expose APIs which may evolve and change over time. This is a long-term challenge if you have to maintain different connectors on your own.

To deal with multi-exchange communication, we implemented the CCXT Library — an open-source standardized interface with which cryptocurrency exchanges are invited to comply. In short, the implementation grants access to most major exchanges.

Sensor Bots

A single sensor bot may interact with all supported crypto exchanges and fetch data from any market. Data completion and integrity are crucial in a trading application, so we put a sizeable effort in building a bullet-proof sensor.

The sensor downloads OHLCV data (open, high, low, and close prices, and volume) of one-minute candles and outputs a data product that is later used as input for other bots.

Some exchanges support downloading raw trades and order book data as well.

Indicator Bots

We borrowed the name indicator to describe a type of bot because the main interest of a trading application is to produce Technical Analysis and Technical Studies. However, the infrastructure allows all sorts of calculations.

In Superalgos, an indicator bot takes other bots’ data products as inputs, processes it applying a user-defined algorithm, and outputs a new dataset.

The data-processing logic may be nested in multiple product definitions within an indicator, or even across indicators, producing multiple data products that may feed other processes downstream.

Visual Definitions

Indicator bots may be created and defined from within a visual environment and the only coding required is the actual data building procedure algorithm. The definition of processes, dependencies, records, dataset structure, and so on, is done from within the visual-scripting interface.