Remove the persistence to handle concurrency

Remove the cached file handles, and close/open the log file anew with each
write. Also write to blabla.log (instead of blabla.log.N) and figure out the
next log number when it comes time to rotate/compress the log.
Remove unused code.
Rotate but don't compress
Add batch submission that includes dimensions
Create log files on demand
Don't silently ignore IO errors
Add an external compressor.

Run this on the base data directory to compress all pending full files.
Add timing / throughput numbers
Remove some unused code
Add MPL license blurb