influxdb/influxdb — pull request #678 by hyc (2014-06-22 23:53:45+00:00)
Title: Use Sorted Duplicates

Saves some disk space. There are more opportunities for
optimization down this avenue. This requires a patch to gomdb https://github.com/szferi/gomdb/pull/25
and also latest LMDB rev https://gitorious.org/mdb/mdb/commit/aa89ca31b3a95f266cd2b354037466a2d950bc1f

For the scenario of 100M keys and 500K series, I got best space savings with 6 byte primary key. But if you use the full 8 byte seriesID as the primary key, then you can simply use MDB_NEXT_DUP to retrieve all the records in a series, and save the expensive Compare(RetrievedKey, EndKey) that your current code does to determine if it's reached the end. (You might see better space savings with 8 byte key, if the data set was even larger. In this data set, there weren't enough records per series to justify the overhead.) Saving 6 or 8 bytes per record is pretty good, anyway.

This code uses DUPFIXED, which assumes that all data values are the same size. If that's not true for your real data, you should omit that flag. If you can use it, it saves another 8 bytes per record.

I didn't look at how real InfluxDB data looks, so if these assumptions are totally off base, please forgive me.