The Definitive Guide To InfluxDB In 2019 — devconnected
By: Antoine Solnichkin
Tags: Programming, Database, DevOps, Software Development, Software Engineering

Essentially, it means that for every point that you are able to store, you have a timestamp associated with it.

The great difference between relational databases and time series databases

But.. couldn’t we use a relational database and simply have a column named ‘time’? Oracle for example includes a TIMESTAMP data type that we could use for that purpose.

You could, but that would be inefficient.

a — Why do we need time series databases?

Three words : fast ingestion rate.

Time series databases systems are built around the predicate that they need to ingest data in a fast and efficient way.

Indeed, relational databases do have a fast ingestion rate for most of them, from 20k to 100k rows per second.

However, the ingestion is not constant over time. Relational databases have one key aspect that make them slow when data tend to grow : indexes.

When you add new entries to your relational database, and if your table contains indexes, your database management system will repeatedly re-index your data for it to be accessed in a fast and efficient way.

As a consequence, the performance of your DBMS tend to decrease over time. The load is also increasing over time, resulting in having difficulties to read your data.

Time series database are optimized for a fast ingestion rate. It means that such index systems are optimized to index data that are aggregated over time : as a consequence, the ingestion rate does not decrease over time and stays quite stable, around 50k to 100k lines per second on a single node.

This graph is inspired by :

https://blog.timescale.com/timescaledb-vs-6a696248104e/

b — Specific concepts about time series databases

On top of the fast ingestion rate, time series databases introduce concepts that are very specific to those technologies.

One of them is data retention. In a traditional relational database, data are stored permanently until your decide to drop them yourself.

Given the use-cases of time series databases, you may want not to keep your data for too long : either because it is too expensive to do so, or because you are not that interested in old data.

Systems like InfluxDB can take care of dropping data after a certain time, with a concept called retention policy (explained in details in part two). You can also decide to run continuous queries on live data in order to perform certain operations.

You could find equivalent operations in a relational database, for example ‘jobs’ in SQL that can run on a given schedule.

c — A Whole Different Ecosystem

Time series databases are very different when it comes to the ecosystem that orbits around them. In general, relational databases are surrounded by applications : web applications, softwares that connect to it to retrieve information or add new entries.

Often, a database is associated with one system. Clients connect to a website, that contacts a database in order to retrieve information. TSDB are built for client plurality : you do not have a simple server accessing the database, but a bunch of different sensors (for example) inserting their data at the same time.

As a consequence, tools were designed in order to have efficient ways to produce data or to consume it.

Data consumption

Data consumption is often done via monitoring tools such as Grafana or Chronograf. Those solutions have built-in solutions to visualize data and even make custom alerts with it.

The data consumers for TSDB

Those tools are often used to create live dashboards that may be graphs, bar charts, gauges or live world maps.

Data Production

Data production is done by agents that are responsible for targeting special elements in your infrastructure and extract metrics from them. Such agents are called “ monitoring agents”. You can easily configure them to query your tools on a given time span. Examples are Telegraf (which is an official monitoring agent), CollectD or StatsD

The data producers for TSDB

Now that you have a better understanding of what time series databases are and how they differ from relational databases, it is time to dive into the specific concepts of InfluxDB.

Module 2 — InfluxDB Concepts Explained

In this section, we are going to explain the key concepts behind InfluxDB and the key query associated with it. InfluxDB embeds its own query language and I think that this point deserves a small explanation.

a — InfluxDB Query Language

Before starting, it is important for you to know which version of InfluxDB you are currently using. As of April 2019, InfluxDB comes in two versions : v1.7+ and v2.0.

v2.0 is currently in alpha version and puts the Flux language as a centric element of the platform. v1.7 is equipped with InfluxQL language (and Flux if you activate it).

Right now, I do recommend to keep on using InfluxQL as Flux is not completely established in the platform.

InfluxQL is a query language that is very similar to SQL and that allows any user to query its data and filter it. Here’s an example of an InfluxQL query :

In the following sections, we are going to explore InfluxDB key concepts, provided with the associated IQL (short for InfluxQL) queries.

b — InfluxDB Key Concepts Explained

In this section, we will go through the list of essential terms to know to deal with InfluxDB in 2019.

Database

A database is a fairly simple concept to understand on its own because you are used to use this term with relational databases. In a SQL environment, a database would host a collection of tables, and even schemas and would represent one instance on its own.

In InfluxDB, a database host a collection of measurements. However, a single InfluxDB instance can host multiple databases. This is where it differs from traditional database systems. This logic is detailed in the graph below :

The most common ways to interact with databases are either creating a database or by navigating into a database in order to see collections (you have to be “in a database” in order to query collections, otherwise it won’t work).

Most used Influx database queries

Measurement

As shown in the graph above, a database stores multiple measurements. You could think of a measurement as a SQL table. It stores data, and even meta data, over time. Data that are meant to coexist together should be stored in the same measurement.

Measurement example

Measurement IFQL example

In a SQL world, data are stored in columns, but in InfluxDB we have two other terms : tags & fields.

Tags & Fields

Warning!

This is a very important chapter as it explains the subtle difference between tags & fields.

When I first started with InfluxDB, I had a hard time grasping exactly why are tags & fields different. For me, they represented ‘columns’ where you could store exactly the same data.

When defining a new ‘column’ in InfluxDB, you have the choice to either declare it as a tag or as a value and it makes a very big difference.

In fact, the biggest difference between the two is that tags are indexed and values are not. Tags can be seen as metadata defining our data in the measurement. They are hints giving additional information about data, but not data itself.

Fields, on the other side, is literally data. In our past example, the temperature ‘column’ would be a field.

Back to our cpu_metrics example, let’s say that we wanted to add a column named ‘location’ as its name states, defines where the sensor is.

Should we add it as a tag or a field?