ukhas/habitat — Issue #159: Reconsider payload telemetry document IDs
Author: adamgreig
Created: 2011-12-19 21:39:19+00:00
State: open
Labels: couchdb

Accorind to http://wiki.apache.org/couchdb/Performance we are basically doing almost the worst possible thing (non-sequential, entirely random, long) for our most common (payload telemetry) type of document.

However, hashing still has a whole ton of advantages -- it's the only way we can be sure to avoid multiple documents for the same received data and works very well with the MVCC system in Couch. Using Couch IDs would require racey database lookups to see if someone has already made the document you want to save; using something generated server-side has similar issues if we want it to be monotonic. Either way we lose the ability to generate the ID given the data.

One easy change would be to use base64 encoded IDs instead of hexadecimal IDs: on sha256, this moves us from 64 bytes to 44 bytes (a ~31% reduction). We could also move to MD5 instead of SHA256, giving 24 byte IDs (a 63% reduction) that would probably still be sufficiently unique (approximately 5E21 documents until 50% chance of collision -- millions of years of telemetry).

However, the changeover would be somewhat traumatic. We'd have to rewrite the current transition stuff, the new dl-fldigi stuff, anything else that relies on the assumption that the docs will be sha256 sums, and then move over all the telem in the database. Not impossible, but not a thing to be done trivially, either.