InfluxDB Data Retention. How to down-sample data and keep it for... | by Shawn Stafford | Towards Data Science

Time series data is often used by operations teams to investigate performance issues. Since we cannot anticipate the source of a performance issue ahead of time, we often err on the side of caution and collect as much data as possible as frequently as possible. This makes it easy to get a very granular picture of what was happening in an environment within the last hour, day, or week. However, keeping more than a week or two of data can become difficult as the number of hosts and volume of data increases.
There are many reasons why it may be necessary to retain data for longer periods of time. It is often useful to refer to data that is weeks, months, or even years old in the following cases:
Capacity planning — by reviewing resource utilization over the past 12 to 24 months, you can project usage forward in order to forecast your budget requirements for the next fiscal year.
Performance degradation — users often report that application performance “seems slower” than it did a few weeks or months ago. Having historical data at your fingertips makes it possible to quantify how much longer tasks are taking or whether there is a correlation between increased load and slower performance.
When dealing with time series data, you will inevitably reach a point where you cannot retain all of the data at full granularity for an indefinite amount of time. A trade-off must be made between how long you retain the data, how much data you retain, and how granular the data is. This article covers how to implement a data retention policy in InfluxDB to ensure that you can down-sample your data, retain it for a specified duration, and perform regular backups to avoid data loss.
For the purpose of this example, we will focus on CPU utilization metrics being collected by a collectd system daemon running on 3 hosts. In this scenario, collectd is submitting CPU data (“load_shortterm”) at 10 second intervals from every host being monitored. The data is being sent directly to InfluxDB using the collectd daemon and stored in a “metrics” database which is defined with a 7-day retention policy. The goal is to retain this data in a “longterm” database with a retention policy of 3 years.
InfluxDB does not listen for collectd input by default. In order to allow data to be submitted by a collectd agent, the InfluxDB server must be configured to listen for collectd connections. This section describes how to configure collectd on a RHEL/CentOS system.
The first step is to create a database on the InfluxDB server to store the incoming collectd data for 7 days. To do this, open a terminal window on the InfluxDB server and use the influx command to connect to the server. Run the following command to create a new database:
CREATE DATABASE metrics WITH DURATION 7d
The next step is to install collectd on the InfluxDB server so that the types.db specification file is available to InfluxDB:
# Install the collectd RPM (available from the EPEL repo)yum install collectd# Locate the types.db file installed by the RPMrpm -ql collectd | grep types.db
Update the InfluxDB config file (/etc/influxdb/influxdb.conf) to listen for collectd data and then restart the influxd service:
[[collectd]]  enabled = true  bind-address = ":8096"  database = "metrics"  typesdb = "/usr/share/collectd/types.db"
Once InfluxDB is listening for collectd input, you will need to install the collectd agent on each of the 3 hosts and configure it to send data to your InfluxDB server.
If you’re not familiar with the InfluxDB data model, figuring out how to locate the data can be the first challenge. Each type of data being collected is referred to as a “measurement” and each measurement can have any number of “tags” associated with it. In the case of data reported by the collectd agent, a measurement will exist for each type of data you have configured the agent to report.
The following commands can be executed in the influx command window to query the database and identify the measurements that will be used in later steps to define a continuous query.
# Switch to the InfluxDB database containing the CollectD dataUSE metrics# Display a list of metrics in the databaseSHOW MEASUREMENTS# Display the tags (keys) used to uniquely identify CPU loadSHOW SERIES FROM load_shortterm
The SHOW SERIES command should produce output similar to the following:
load_shortterm,host=host1,type=loadload_shortterm,host=host2,type=loadload_shortterm,host=host3,type=load
There are many options for defining a long-term retention policy in InfluxDB. One option is to create a new retention policy within the existing metrics database and store the long-term data alongside your short-term data:
CREATE RETENTION POLICY longterm_policy ON metrics DURATION 156w REPLICATION 1
However, I would avoid creating multiple retention policies within a single database unless you have a particular justification for doing so. If there are multiple retention policies within a single database, your queries will need to reference the retention policy explicitly (database.policy.measurement) when operating on the data. This will make your queries more fragile because of the explicit references to a retention policy.
In addition, the incoming data in the metrics database is high-volume transient data. It would be very expensive to back up the entire database and the impact of losing 1 week of very granular data would not be worth the effort.
A better approach would be to define a separate database specifically for any long-term data you wish to retain. Doing so will make it easy to change the retention policy later or back up the entire database without affecting any external reports or queries. The following example shows how to create a database with a retention policy of 3 years (156 weeks):
CREATE DATABASE longterm WITH DURATION 156w
Create a continuous query to down-sample the data from 10 second intervals down to 15 minute intervals:
CREATE CONTINUOUS QUERY aggregate_load ON longtermBEGIN  SELECT max(value) AS value   INTO longterm.autogen.load_shortterm   FROM metrics.autogen.load_shortterm   GROUP BY time(15m),* END
Important Notes:
The aggregation function you choose will depend on your use case. The max function is being used in this case because we typically care about the peak load average during an interval. Using a mean calculation would dilute these spikes and make it harder to identify short spikes in activity.
The INTO and FROM clause require a fully qualified measurement (database.policy.measurement). If you define the default retention policy as part of the database creation as shown in the examples above, the retention policy will be called “autogen” by default.
The GROUP BY clause contains a * wildcard, which means “all tags.” This ensures that the aggregation is performed across only the data points where all tags are identical. Without that wildcard (or an explicit list of tags), the aggregation would be performed across all data for all hosts, resulting in a single meaningless aggregation. In most cases a wildcard should be used in the GROUP BY to ensure that aggregation is performed on the same unique item.
The multi-line formatting shown above is for readability. You’ll need to structure the command on a single line when executing it within the influx command.
Once you have successfully created a continuous query, you should see measurements appear at the end of the first time interval. This can be verified by listing the measurements in the same way you discovered them previously:
USE longtermSHOW MEASUREMENTSSHOW SERIES FROM load_shortterm
The Grafana charts below show the difference between raw data (10 second intervals) and aggregated data (15 minute intervals).
InfluxDB comes with a command-line utility for performing database backups. Execution is reasonably straight forward:
influxd backup -portable -database longterm /backup/longterm
Additional arguments can be specified to restrict the export by a specific retention policy, shard, or date range.
By default, the InfluxDB service writes its logging output to /var/log/messages. Several lines of output are generated each time a continuous query is run. The following command can be used to look for continuous query execution messages:
tail -f /var/log/messages | grep "Finished continuous"
The matching lines will look something like this:
Jan  7 15:00:00 influxdbhost01 influxd:    ts=2019-01-07T20:00:00.127002Z    lvl=info    msg="Finished continuous query"    log_id=0BoYFF20000    service=continuous_querier    trace_id=0CrUFfTl000    op_name=continuous_querier_execute    name=aggregate_load    db_instance=longterm    written=129    start=2019-01-07T19:59:00.000000Z    end=2019-01-07T20:00:00.000000Z    duration=7ms
The log output contains several important pieces of information:
name — the name of the continuous query
written — the number of measurement records that were written
duration — how long the query took to execute
Obviously it would be a bad idea to define a continuous query that is scheduled to run more frequently than the time it takes to complete the query.
Continuous queries are a convenient way to selectively down-sample data and retain it for longer periods of time. By creating thoughtful retention policies and backup procedures, it is possible to retain historical time series data for months or even years. Making this historical data readily accessible within the same visualization interface as your real-time metrics will empower application owners and operations staff to make informed decisions based on historical context and trends, rather than relying on a “gut feel” approach to budgeting, capacity planning, or problem investigation.