cfpb/qu — pull request #57 by cndreisbach (2013-07-16 02:10:24+00:00)
Title: Key compression

Compress keys on storage and decompress them on retrieval. This has several advantages for us:

* Records take up less room. Without key compression, average record size is 2313 bytes. With key compression, it is 948 bytes. That is only **41 percent of the original size of the record.** This can't be overstated: that's less data in memory, which allows us to serve the HMDA data with less resources.

* Loading is actually faster. We are doing more to the data, but I/O is our real problem, and by writing less data, we speed things up. On my local machine, it went from 19.8 minutes to load a 1% of the HMDA data to 17.6 minutes, which isn't a huge jump, but still, better than nothing. Indexing still takes a very long time and is a large portion of that time.